Registration based addressing over multiple networks with digital audio communication

ABSTRACT

A multimedia collaboration system that integrates separate real-time and asynchronous networks—the former for real-time audio and video, and the latter for control signals and textual, graphical and other data—in a manner that is interoperable across different computer and network operating system platforms and which closely approximates the experience of face-to-face collaboration, while liberating the participants from the limitations of time and distance. These capabilities are achieved by exploiting a variety of hardware, software and networking technologies in a manner that preserves the quality and integrity of audio/video/data and other multimedia information, even after wide area transmission, and at a significantly reduced networking cost as compared to what would be required by presently known approaches. The system architecture is readily scalable to the largest enterprise network environments. It accommodates differing levels of collaborative capabilities available to individual users and permits high-quality audio and video capabilities to be readily superimposed onto existing personal computers and workstations and their interconnecting LANs and WANs. In a particular preferred embodiment, a plurality of geographically dispersed multimedia LANs are interconnected by a WAN. The demands made on the WAN are significantly reduced by employing multi-hopping techniques, including dynamically avoiding the unnecessary decompression of data at intermediate hops, and exploiting video mosaicing, cut-and-paste and audio mixing technologies so that significantly fewer wide area transmission paths are required while maintaining the high quality of the transmitted audio/video.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of U.S. patent applicationSer. No. 09/702,737 filed Nov. 1, 2000, which is a Continuation of U.S.patent application Ser. No. 08/994,848, filed Dec. 19, 1997, now U.S.Pat. No. 6,237,025, which is a continuation of U.S. patent applicationSer. No. 08/660,461, filed Jun. 7, 1996, now U.S. Pat. No. 5,802,294,which is a continuation of U.S. patent application Ser. No. 08/131,523,filed Oct. 1, 1993, now U.S. Pat. No. 5,689,641, the disclosures ofwhich are incorporated herein by reference.

This application is also related to U.S. patent application Ser. No.______ filed Oct. 28, 2005 (Attorney Docket No. CA1420); U.S. patentapplication Ser. No. ______ filed Oct. 28, 2005 (Attorney Docket No.CA1421); U.S. patent application Ser. No. ______, filed Oct. 28, 2005(Attorney Docket No. CA1422); and U.S. patent application Ser. No.______ filed Oct. 28, 2005 (Attorney Docket No. CA1424).

This application is also related to U.S. patent application Ser. No.10/721,343, filed Nov. 26, 2003; U.S. patent application Ser. No.10/722,051 filed Nov. 26, 2003; U.S. patent application Ser. No.10/721,905 filed Nov. 26, 2003; U.S. patent application Ser. No.10/721,385 filed Nov. 26, 2003.

BACKGROUND OF THE INVENTION

The present invention relates to computer-based systems for enhancingcollaboration between and among individuals who are separated bydistance and/or time (referred to herein as “distributedcollaboration”). Principal among the invention's goals is to replicatein a desktop environment, to the maximum extent possible, the fullrange, level and intensity of interpersonal communication andinformation sharing which would occur if all the participants weretogether in the same room at the same time (referred to herein as“face-to-face collaboration”).

It is well known to behavioral scientists that interpersonalcommunication involves a large number of subtle and complex visual cues,referred to by names like “eye contact” and “body language,” whichprovide additional information over and above the spoken words andexplicit gestures. These cues are, for the most part, processedsubconsciously by the participants, and often control the course of ameeting.

In addition to spoken words, demonstrative gestures and behavioral cues,collaboration often involves the sharing of visual information—e.g.,printed material such as articles, drawings, photographs, charts andgraphs, as well as videotapes and computer-based animations,visualizations and other displays—in such a way that the participantscan collectively and interactively examine, discuss, annotate and revisethe information. This combination of spoken words, gestures, visual cuesand interactive data sharing significantly enhances the effectiveness ofcollaboration in a variety of contexts, such as “brainstorming” sessionsamong professionals in a particular field, consultations between one ormore experts and one or more clients, sensitive business or politicalnegotiations, and the like. In distributed collaboration settings, then,where the participants cannot be in the same place at the same time, thebeneficial effects of face-to-face collaboration will be realized onlyto the extent that each of the remotely located participants can be“recreated” at each site.

To illustrate the difficulties inherent in reproducing the beneficialeffects of face-to-face collaboration in a distributed collaborationenvironment, consider the case of decision-making in the fast-movingcommodities trading markets, where many thousands of dollars of profit(or loss) may depend on an expert trader making the fight decisionwithin hours, or even minutes, of receiving a request from a distantclient. The expert requires immediate access to a wide range ofpotentially relevant information such as financial data, historicalpricing information, current price quotes, newswire services, governmentpolicies and programs, economic forecasts, weather reports, etc. Much ofthis information can be processed by the expert in isolation. However,before making a decision to buy or sell, he or she will frequently needto discuss the information with other experts, who may be geographicallydispersed, and with the client. One or more of these other experts maybe in a meeting, on another call, or otherwise temporarily unavailable.In this event, the expert must communicate “asynchronously”—to bridgetime as well as distance.

As discussed below, prior art desktop videoconferencing systems provide,at best, only a partial solution to the challenges of distributedcollaboration in real time, primarily because of their lack ofhigh-quality video (which is necessary for capturing the visual cuesdiscussed above) and their limited data sharing capabilities. Similarly,telephone answering machines, voice mail, fax machines and conventionalelectronic mail systems provide incomplete solutions to the problemspresented by deferred (asynchronous) collaboration because they aretotally incapable of communicating visual cues, gestures, etc and, likeconventional videoconferencing systems, are generally limited in therichness of the data that can be exchanged.

It has been proposed to extend traditional videoconferencingcapabilities from conference centers, where groups of participants mustassemble in the same room, to the desktop, where individual participantsmay remain in their office or home. Such a system is disclosed in U.SPat. No. 4,710,917 to Tompkins et al for Video Conferencing Networkissued on Dec. 1, 1987. It has also been proposed to augment such videoconferencing systems with limited “video mail” facilities. However, suchdedicated videoconferencing systems (and extensions thereof) do noteffectively leverage the investment in existing embedded informationinfrastructures—such as desktop personal computers and workstations,local area network (LAN) and wide area network (WAN) environments,building wiring, etc.—to facilitate interactive sharing of data in theform of text, images, charts, graphs, recorded video, screen displaysand the like. That is, they attempt to add computing capabilities to avideoconferencing system, rather than adding multimedia andcollaborative capabilities to the user's existing computer system. Thus,while such systems may be useful in limited contexts, they do notprovide the capabilities required for maximally effective collaboration,and are not cost-effective.

Conversely, audio and video capture and processing capabilities haverecently been integrated into desktop and portable personal computersand workstations (hereinafter generically referred to as“workstations”). These capabilities have been used primarily in desktopmultimedia authoring systems for producing CD-ROM-based works. Whilesuch systems are capable of processing, combining, and recording audio,video and data locally (i.e., at the desktop), they do not adequatelysupport networked collaborative environments, principally due to thesubstantial bandwidth requirements for real-time transmission ofhigh-quality, digitized audio and full-motion video which precludeconventional LANs from supporting more than a few workstations. Thus,although currently available desktop multimedia computers frequentlyinclude videoconferencing and other multimedia or collaborativecapabilities within their advertised feature set (see, e.g., AReinhardt, “Video Conquers the Desktop,” BYTE, September 1993, pp.64-90), such systems have not yet solved the many problems inherent inany practical implementation of a scalable collaboration system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an enterprise view of a desk-top collaboration systemembodiment of the present invention.

FIGS. 2A and 2B are photographs which attempt to illustrate, to theextent possible in a still image, the high-quality of the full-motionvideo and related user interface displays that appear on typical CMWscreens which may be generated during operation of a preferredembodiment of the invention.

FIG. 3 is a block and schematic diagram of a preferred embodiment of a“multimedia local area network” (MLAN) in accordance with a desktopcollaboration system embodiment of the present invention.

FIG. 4 is a block and schematic diagram illustrating how a plurality ofgeographically dispersed MLANs of the type shown in FIG. 3 can beconnected via a wide area network in accordance with the presentinvention.

FIG. 5 is a schematic diagram illustrating how collaboration sites atdistant locations L1-L8 are conventionally interconnected over a widearea network by individually connecting each site to every other site.

FIG. 6 is a schematic diagram illustrating how collaboration sites atdistant locations L1-L8 are interconnected over a wide area network in apreferred embodiment of the invention using a multi-hopping approach.

FIG. 7 is a block diagram illustrating a preferred embodiment of videomosaicing circuitry provided in the MLAN of FIG. 3.

FIGS. 8A, 8B and 8C illustrate the video window on a typical CMW screenwhich may be generated during operation of a preferred embodiment of thepresent invention, and which contains only the callee for two-partycalls (8A) and a video mosaic of all participants, e.g., for four-party(8B) or eight-party (8C) conference calls.

FIG. 9 is a block diagram illustrating a preferred embodiment of audiomixing circuitry provided in the MLAN of FIG. 3.

FIG. 10 is a block diagram illustrating video cut-and-paste circuitryprovided in the MLAN of FIG. 3.

FIG. 11 is a schematic diagram illustrating typical operation of thevideo cut-and-paste circuitry in FIG. 10.

FIGS. 12-17 (consisting of FIGS. 12A, 12B, 13A, 13B, 14A, 14B, 15A, 15B,16, 17A and 17B) illustrate various examples of how a preferredembodiment of the present invention provides video mosaicing, videocut-and-pasting, and audio mixing at a plurality of distant sites fortransmission over a wide area network in order to provide, at the CMW ofeach conference participant, video images and audio captured from theother conference participants.

FIGS. 18A and 18B illustrate various preferred embodiments of a CMWwhich may be employed in accordance with the present invention.

FIG. 19 is a schematic diagram of a preferred embodiment of a CMW add-onbox containing integrated audio and video L/O circuitry in accordancewith the present invention.

FIG. 20 illustrates CMW software in accordance with a preferredembodiment of the present invention, integrated with standardmultitasking operating system and applications software.

FIG. 21 illustrates software modules which may be provided for runningon the MLAN Server in the MLAN of FIG. 3 for controlling operation ofthe AV and Data Networks.

FIG. 22 illustrates an enlarged example of “speed-dial” face icons ofcertain collaboration participants in a Collaboration Initiator windowon a typical CMW screen which may be generated during operation of apreferred embodiment of the present invention.

FIG. 23 is a diagrammatic representation of the basic operating eventsoccurring in a preferred embodiment of the present invention duringinitiation of a two-party call.

FIG. 24 is a block and schematic diagram illustrating how physicalconnections are established in the MLAN of FIG. 3 for physicallyconnecting first and second workstations for a two-party videoconferencecalls

FIG. 25 is a block and schematic diagram illustrating how physicalconnections are preferably established in MLANs such as illustrated inFIG. 3, for a two-party call between a first CMW located at one site anda second CMW located at a remote site.

FIGS. 26 and 27 are block and schematic diagrams illustrating howconference bridging is preferably provided in the MLAN of FIG. 3.

FIG. 28 diagrammatically illustrates how a snapshot with annotations maybe stored in a plurality of bitmaps during data sharing.

FIG. 29 is a schematic and diagrammatic illustration of the interactionamong multimedia mail (MMM), multimedia call/conference recording (MMCR)and multimedia document management (MMDM) facilities.

FIG. 30 is a schematic and diagrammatic illustration of the multimediadocument architecture employed in a preferred embodiment of theinvention.

FIG. 31A illustrates a centralized Audio/Video Storage Server.

FIG. 31B is a schematic and diagrammatic illustration of theinteractions between the Audio/Video Storage Server and the remainder ofthe CMW System.

FIG. 31C illustrates an alternative embodiment of the interactionsillustrated in FIG. 31B.

FIG. 31D is a schematic and diagrammatic illustration of the integrationof MMM MMCR and MMDM facilities in a preferred embodiment of theinvention.

FIG. 32 illustrates a generalized hardware implementation of a scalableAudio/Video Storage Server.

FIG. 33 illustrates a higher throughput version of the serverillustrated in FIG. 32, using SCSI-based crosspoint switching toincrease the number of possible simultaneous file transfers.

FIG. 34 illustrates the resulting multimedia collaboration environmentachieved by the integration of audio/video/data teleconferencing andMMCR, MMM and MMDM.

FIGS. 35-42 illustrate a series of CMW screens which may be generatedduring operation of a preferred embodiment of the present invention fora typical scenario involving a remote expert who takes advantage of manyof the features provided by the present invention.

SUMMARY OF THE INVENTION

In accordance with the present invention, computer hardware, softwareand communications technologies are combined in novel ways to produce amultimedia collaboration system that greatly facilitates distributedcollaboration, in part by replicating the benefits of face-to-facecollaboration. The system tightly integrates a carefully selected set ofmultimedia and collaborative capabilities, principal among which aredesktop teleconferencing and multimedia mail.

As used herein, desk-top teleconferencing includes real-time audioand/or video teleconferencing, as well as data conferencing Dataconferencing, in turn, includes snapshot sharing (sharing of “snapshots”of selected regions of the user's screen), application sharing (sharedcontrol of running applications), shared whiteboard (equivalent tosharing a “blank” window), and associated telepointing and annotationcapabilities. Teleconferences may be recorded and stored for laterplayback, including both audio/video and all data interactions.

While desktop teleconferencing supports real-time interactions,multimedia mail permits the asynchronous exchange of arbitrarymultimedia documents, including previously recorded teleconferences.Indeed, it is to be understood that the multimedia capabilitiesunderlying desktop teleconferencing and multimedia mail also greatlyfacilitate the creation, viewing, and manipulation of high-qualitymultimedia documents in general, including animations and visualizationsthat might be developed, for example, in the course of informationanalysis and modeling. Further, these animations and visualizations maybe generated for individual rather than collaborative use, such that thepresent invention has utility beyond a collaboration context.

The preferred embodiment of the invention is a collaborative multimediaworkstation (CMW) system wherein very high-quality audio and videocapabilities can be readily superimposed onto an enterprise's existingcomputing and network infrastructure, including workstations, LANs,WANs, and building wiring.

In a preferred embodiment, the system architecture employs separatereal-time and asynchronous networks—the former for real-time audio andvideo, and the latter for non-real-time audio and video, text, graphicsand other data, as well as control signals. These networks areinteroperable across different computers (e.g., Macintosh, Intel-basedPCs, and Sun workstations), operating systems (e.g., Apple System 7,DOS/Windows, and UNIX) and network operating systems (e.g., NovellNetware and Sun ONC+). In many cases, both networks can actually sharethe same cabling and wall jack connector.

The system architecture also accommodates the situation in which theuser's desktop computing and/or communications equipment providesvarying levels of media-handling capability. For example, acollaboration session—whether real-time or asynchronous—may includeparticipants whose equipment provides capabilities ranging from audioonly (a telephone) or data only (a personal computer with a modem) to afull complement of real-time, high-fidelity audio and full-motion video,and high-speed data network facilities.

The CMW system architecture is readily, scalable to very largeenterprise-wide network environments accommodating thousands of users.Further, it is an open architecture that can accommodate appropriatestandards. Finally, the CMW system incorporates an intuitive, yetpowerful, user interface, making the system easy to learn and use.

The present invention thus provides a distributed multimediacollaboration environment that achieves the benefits of face-to-facecollaboration as nearly as possible, leverages (“snaps on to”) existingcomputing and network infrastructure to the maximum extent possible,scales to very large networks consisting of thousand of workstations,accommodates emerging standards, and is easy to learn and use. Thespecific nature of the invention, as well as its objects, features,advantages and uses, will become more readily apparent from thefollowing detailed description and examples, and from the accompanyingdrawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT Overall SystemArchitecture

Referring initially to FIG. 1, illustrated therein is an overalldiagrammatic view of a multimedia collaboration system in accordancewith the present invention. As shown, each of a plurality of “multimedialocal area networks” (MLANs) 10 connects, via lines 13, a plurality ofCMWs 12-1 to 12-10 and provides audio/video/data networking forsupporting collaboration among CMW users WAN 15 in turn connectsmultiple MLANs 10, and typically includes appropriate combinations ofcommon carrier analog and digital transmission networks Multiple MLANs10 on the same physical premises may be connected via bridges/routes 11,as shown, to WANs and one another.

In accordance with the present invention, the system of FIG. 1accommodates both “real time” delay and jitter-sensitive signals (e.g.,real-time audio and video teleconferencing) and classical asynchronousdata (e.g., data control signals as well as shared textual, graphics andother media) communication among multiple CMWs 12 regardless of theirlocation. Although only ten CMWs 12 are illustrated in FIG. 1, it willbe understood that many more could be provided. As also indicated inFIG. 1, various other multimedia resources 16 (e.g., VCRs, laserdiscs,TV feeds, etc.) are connected to MLANs 10 and are thereby accessible byindividual CMWs 12.

CMW 12 in FIG. 1 may use any of a variety of types of operating systems,such as Apple System 7, UNIX, DOS/Windows and OS/2. The CMWs can alsohave different types of window systems. Specific preferred embodimentsof a CMW 12 are described hereinafter in connection with FIGS. 18A and18B. Note that this invention allows for a mix of operating systems andwindow systems across individual CMWs.

In the preferred embodiment, CMW 12 in FIG. 1 provides real-timeaudio/video/data capabilities along with the usual data processingcapabilities provided by its operating system CMW 12 also provides forbidirectional communication, via lines 13, within MLAN 10, foraudio/video signals as well as data signals Audio/video signalstransmitted from a CMW 12 typically comprise a high-quality live videoimage and audio of the CMW operator. These signals are obtained from avideo camera and microphone provided at the CMW (via an add-on unit orpartially or totally integrated into the CMW), processed, and then madeavailable to low-cost network transmission subsystems.

Audio/video signals received by a CMW 12 from MLAN 10 may typicallyinclude: video images of one or more conference participants andassociated audio, video and data from multimedia mail, previouslyrecorded audio/video from previous calls and conferences, and standardbroadcast television (e.g., CNN). Received video signals are displayedon the CMW screen or on an adjacent monitor, and the accompanying audiois reproduced by a speaker provided in or near the CMW. In general, therequired transducers and signal processing hardware could be integratedinto the CMW, or be provided via a CMW add-on unit, as appropriate.

In the preferred embodiment, it has been found particularly advantageousto provide the above-described video at standard NTSC-quality TVperformance (i.e., 30 frames per second at 640×480 pixels per frame andthe equivalent of 24 bits of color per pixel) with accompanyinghigh-fidelity audio (typically between 7 and 15 KHz). For example, FIG.2A illustrates a CMW screen containing live, full-motion video of threeconference participants, while FIG. 2B illustrates data shared andannotated by those conferees (lower left window).

Multimedia Local Area Network

Referring next to FIG. 3, illustrated therein is a preferred embodimentof MLAN 10 having ten CMWs (12-1, 12-2,-12-10), coupled therein vialines 13 a and 13 b MLAN 10 typically extends over a distance from a fewhundred feet to a few miles, and is usually located within a building ora group of proximate buildings.

Given the current state of networking technologies, it is useful (forthe sake of maintaining quality and minimizing costs) to provideseparate signal paths for real-time audio/video and classicalasynchronous data communications (including digitized audio and videoenclosures of multimedia mail messages that are free from real-timedelivery constraints). At the moment, analog methods for carryingreal-time audio/video are preferred. In the future, digital methods maybe used. Eventually, digital audio and video signal paths may bemultiplexed with the data signal path as a common digital stream.Another alternative is to multiplex real-time and asynchronous datapaths together using analog multiplexing methods. For the purposes ofthe present application, however, we will treat these two signal pathsas using physically separate wires. Further, as the current preferredembodiment uses analog networking for audio and video, it alsophysically separates the real-time and asynchronous switching vehiclesand, in particular, assumes an analog audio/video switch. In the future,a common switching vehicle (e.g., ATM) could be used.

The MLAN 10 thus can be implemented in the preferred embodiment usingconventional technology, such as typical Data LAN hubs 25 and A/VSwitching Circuitry 30 (as used in television studios and otherclosed-circuit television networks), linked to the CMWs 12 viaappropriate transceivers and unshielded twisted pair (UTP) wiring. Notein FIG. 1 that lines 13, which interconnect each CMW 12 within itsrespective MLAN 10, comprise two sets of lines 13 a and 13 b Lines 13 aprovide bidirectional communication of audio/video within MLAN 10, whilelines 13 b provide for the bidirectional communication of data. Thisseparation permits conventional LANs to be used for data communicationsand a supplemental network to be used for audio/video communications.Although this separation is advantageous in the preferred embodiment, itis again to be understood that audio/video/data networking can also beimplemented using a single pair of lines for both audio/video and datacommunications via a very wide variety of analog and digitalmultiplexing schemes.

While lines 13 a and 13 b may be implemented in various ways, it iscurrently preferred to use commonly installed 4-pair UTP telephonewires, wherein one pair is used for incoming video with accompanyingaudio (mono or stereo) multiplexed in, wherein another pair is used foroutgoing multiplexed audio/video, and wherein the remaining two pairsare used for carrying incoming and outgoing data in ways consistent withexisting LANs. For example, 10 BaseT Ethernet uses RJ-45 pins 1, 2, 4,and 6, leaving pins 3, 5, 7, and 8 available for the two A/V twistedpairs. The resulting system is compatible with standard (AT&T 258A,EIA/TIA 568, 8P8C, 10 BaseT, ISDN, 6P6C, etc.) telephone wiring foundcommonly throughout telephone and LAN cable plants in most officebuildings throughout the world. These UTP wires are used in a hierarchyor peer arrangements of star topologies to create MLAN 10, describedbelow. Note that the distance range of the data wires often must matchthat of the video and audio Various UTP-compatible data. LAN networksmay be used, such as Ethernet, token ring, FDDI, ATM etc. For distanceslonger than the maximum distance specified by the data LAN protocol,data signals can be additionally processed for proper UTP operations.

As shown in FIG. 3, lines 13 a from each CMW 12 are coupled to aconventional Data LAN hub 25, which facilitates the communication ofdata (including control signals) among such CMWs Lines 13 b in FIG. 3are connected to A/V Switching Circuitry 30. One or more conferencebridges 35 are coupled to ANV Switching Circuitry 30 and possibly (ifneeded) the Data LAN hub 25, via lines 35 b and 35 a, respectively, forproviding multi-party Conferencing in a particularly advantageousmanner, as will hereinafter be described in detail A WAN gateway 40provides for bidirectional communication between MLAN 10 and WAN 15 inFIG. 1. For this purpose, Data LAN hub 25 and A/V Switching Circuitry 30are coupled to WAN gateway 40 via outputs 25 a and 30 a, respectively.Other devices connect to the ANV Switching Circuitry 30 and Data LAN hub25 to add additional features, such as multimedia mail, conferencerecording, etc.) as discussed below.

Control of A/V Switching Circuitry 30, conference bridges 35 and WANgateway 40 in FIG. 3 is provided by MLAN Server 60 via lines 60 b, 60 c,and 60 d, respectively. In a preferred embodiment, MLAN Server 60supports the TCP/IP network protocol suite. Accordingly, softwareprocesses on CMWs 12 communicate with one another and MLAN Server 60 viaMLAN 10 using these protocols. Other network protocols could also beused, such as IPX. The manner in which software running on MLAN Server60 controls the operation of MLAN 10 will be described in detailhereinafter.

Note in FIG. 3 that Data LAN hub 25, A/V Switching Circuitry 30 and MLANServer 60 also provide respective lines 25 b, 30 b, and 60 e forcoupling to additional multimedia resources 16 (FIG. 1), such asmultimedia document management, multimedia databases, radio/TV channelsetc Data LAN hub 25 (via bridges/routers 11 in FIG. 1) and A/VSwitching Circuitry 30 additionally provide lines 25 c and 30 c forcoupling to one or more other MLANs 10 which may be in the same locality(i.e., not far enough away to require use of WAN technology) Where WANsare required, WAN gateways 40 are used to provide highest qualitycompression methods and standards in a shared resource fashion, thusminimizing costs at the workstation for a given WAN quality level, asdiscussed below.

The basic operation of the preferred embodiment of the resultingcollaboration system shown in FIGS. 1 and 3 will next be considered.Important features of the present invention reside in providing not onlymulti-party real-time desktop audio/video/data teleconferencing amonggeographically distributed CMWs, but also in providing from the samedesktop audio/video/data/text/graphics mail capabilities, as well asaccess to other resources, such as databases, audio and video files,overview cameras, standard TV channels, etc FIG. 2B illustrates a CMWscreen showing a multimedia EMAIL mailbox (top left window) containingreferences to a number of received messages along with a video enclosure(top right window) to the selected message.

A/V Switching Circuitry 30 (whether digital or analog as in thepreferred embodiment) provides common audio/video switching for CMWs 12,conference bridges 35, WAN gateway 40 and multimedia resources 16, asdetermined by MLAN Server 60, which in turn controls conference bridges35 and WAN gateway 44). Similarly, asynchronous data is communicatedwithin MLAN 10 utilizing common data communications formats wherepossible (e.g., for snapshot sharing) so that the system can handle suchdata in a common manner, regardless of origin, thereby facilitatingmultimedia mail and data sharing as well as audio/video communications.

For example, to provide multi-party teleconferencing, an initiating CMW12 signals MLAN Server 60 via Data LAN hub 25 identifying the desiredconference participants. After determining which of these conferees willaccept the call, MLAN Server 60 controls A/V Switching Circuitry 30 (andCMW software via the data network) to set up the required audio/videoand data paths to conferees at the same location as the initiating CMW.

When one or more conferees are at distant locations, the respective MLANServers 60 of the involved MLANs 10, on a peer-to-peer basis, controltheir respective A/V Switching Circuitry 30, conference bridges 35, andWAN gateways 40 to set up appropriate communication paths (via WAN 15 inFIG. 1) as required for interconnecting the conferees MLAN Servers 60also communicate with one another via data paths so that each MLAN 10contains updated information as to the capabilities of all of the systemCMWs 12, and also the current locations of all parties available forteleconferencing.

The data conferencing component of the above-described system supportsthe sharing of visual information at one or more CMWs (as described ingreater detail below). This encompasses both “snapshot sharing” (sharing“snapshots” of complete or partial screens, or of one or more selectedwindows) and “application sharing” (sharing both the control and displayof running applications) When transferring images, lossless or slightlylossy image compression can be used to reduce network bandwidthrequirements and user-perceived delay while maintaining high imagequality.

In all cases, any participant can point at or annotate the shared data.These associated telepointers and annotations appear on everyparticipant's CMW screen as they are drawn (i.e., effectively in realtime). For example, note FIG. 2B which illustrates a typical CMW screenduring a multi-party teleconferencing session, wherein the screencontains annotated shared data as well as video images of the conferees.As described in greater detail below, all or portions of the audio/videoand data of the teleconference can be recorded at a CMW (or within MLAN10), complete with all the data interactions.

In the above-described preferred embodiment, audio/video file servicescan be implemented either at the individual CMWs 12 or by employing acentralized audio/video storage server. This is one example of the manytypes of additional servers that can be added to the basic system ofMLANs 10. A similar approach is used for incorporating other multimediaservices, such as commercial TV channels, multimedia mail, multimediadocument management, multimedia conference recording, visualizationservers, etc. (as described in greater detail below) Certainly,applications that run self-contained on a CMW can be readily added, butthe invention extends this capability greatly in the way that MLAN 10,storage and other functions are implemented and leveraged.

In particular, standard signal formats, network interfaces, userinterface messages, and call models can allow virtually any multimediaresource to be smoothly integrated into the system. Factors facilitatingsuch smooth integration include: (i) a common mechanism for user accessacross the network; (ii) a common metaphor (e.g., placing a call) forthe user to initiate use of such resource; (iii) the ability for onefunction (e.g., a multimedia conference or multimedia database) toaccess and exchange information with another function (e.g., multimediamail); and (iv) the ability to extend such access of one networkedfunction by another networked function to relatively complex nestings ofsimpler functions (for example, record a multimedia conference in whicha group of users has accessed multimedia mail messages and transferredthem to a multimedia database, and then send part of the conferencerecording just created as a new multimedia mail message, utilizing amultimedia mail editor if necessary).

A simple example of the smooth integration of functions made possible bythe above-described approach is that the GUI and software used forsnapshot sharing (described below) can also be used as an input/outputinterface for multimedia mail and more general forms of multimediadocuments. This can be accomplished by structuring the interprocesscommunication protocols to be uniform across all these applications.More complicated examples—specifically multimedia conference recording,multimedia mail and multimedia document management—will be presented indetail below.

Wide Area Network

Next to be described in connection with FIG. 4 is the advantageousmanner in which the present invention provides for real-timeaudio/video/data communication among geographically dispersed MLANs 10via WAN 15 (FIG. 1), whereby communication delays, cost and degradationof video quality are significantly minimized from what would otherwisebe expected.

Four MLANs 10 are illustrated at locations A, B, C and D CMWs 12-1 to12-10, A/V Switching Circuitry 30, Data LAN hub 25, and WAN gateway 40at each location correspond to those shown in FIGS. 1 and 3 WAN gateway40 in FIG. 4 will be seen to comprise a router/codec (R&C) bank 42coupled to WAN 15 via WAN switching multiplexer 44. The router is usedfor data interconnection and the codec is used for audio/videointerconnection (for multimedia mail and document transmission, as wellas videoconferencing). Codecs from multiple vendors, or supportingvarious compression algorithms may be employed. In the preferredembodiment, the router and codec are combined with the switchingmultiplexer to form a single integrated unit.

Typically, WAN 15 is comprised of T1 or ISDN common-carrier-provideddigital links (switched or dedicated), in which case WAN switchingmultiplexers 44 are of the appropriate type (T1, ISDN, fractional T1,T3, switched 56 Kbps, etc.). Note that the WAN switching multiplexer 44typically creates subchannels whose bandwidth is a multiple of 64 Kbps(i.e., 256 Kbps, 384, 768, etc.) among the T1, T3 or ISDN carriers.Inverse multiplexers may be required when using 56 Kbps dedicated orswitched services from these carriers.

In the MLAN 10 to WAN 15 direction, router/codec bank 42 in FIG. 4provides conventional analog-to-digital conversion and compression ofaudio/video signals received from A/V Switching Circuitry 30 fortransmission to WAN 15 via WAN switching multiplexer 44, along withtransmission and routing of data signals received from Data LAN hub 25.In the WAN 15 to MLAN 10 direction, each router/codec bank 42 in FIG. 4provides digital-to-analog conversion and decompression of audio/videodigital signals received from WAN 15 via WAN switching multiplexer 44for transmission to A/V Switching Circuitry 30, along with thetransmission to Data LAN hub 25 of data signals received from WAN 15.

The system also provides optimal routes for audio/video signals throughthe WAN. For example, in FIG. 4, location. A can take either a directroute to location D via path 47, or a two-hop route through location Cvia paths 48 and 49. If the direct path 47 linking location A andlocation D is unavailable, the multipath route via location C and paths48 and 49 could be used.

In a more complex network, several multi-hop routes are typicallyavailable, in which case the routing system handles the decision making,which for example can be based on network loading considerations. Notethe resulting two-level network hierarchy: a MLAN 10 to MLAN 10 (i.e.,site-to-site) service connecting codecs with one another only atconnection endpoints.

The cost savings made possible by providing the above-describedmulti-hop capability (with intermediate codec bypassing) are verysignificant as will become evident by noting the examples of FIGS. 5 and6. FIG. 5 shows that using the conventional “fully connected mesh”location-to-location approach, thirty-six WAN links are required forinterconnecting the nine locations L1 to L8. On the other hand, usingthe above multi-hop capabilities, only nine WAN links are required, asshown in FIG. 6. As the number of locations increase, the difference incost becomes even greater, growing as the square of the number of sites.For example, for 100 locations, the conventional approach would requireabout 5,000 WAN links, while the multi-hop approach of the presentinvention would typically require 300 or fewer (possibly considerablyfewer) WAN links. Although specific WAN links for the multi-hop approachof the invention would require higher bandwidth to carry the additionaltraffic, the cost involved is very much smaller as compared to the costfor the very much larger number of WAN links required by theconventional approach.

At the endpoints of a wide-area call, the WAN switching multiplexerroutes audio/video signals directly from the WAN network interfacethrough an available codec to MLAN 10 and vice versa. At intermediatehops in the network, however, video signals are routed from one networkinterface on the WAN switching multiplexer to another network interface.Although A/V Switching Circuitry 30 could be used for this purpose, thepreferred embodiment provides switching functionality inside the WANswitching multiplexer. By doing so, it avoids having to routeaudio/video signals through codecs to the analog switching circuitry,thereby avoiding additional codec delays at the intermediate locations.

A product capable of performing the basic switching functions describedabove for WAN switching multiplexer 44 is available from TeleosCorporation, Eatontown, N.J. This product is not known to have been usedfor providing audio/video multi-hopping and dynamic switching amongvarious WAN links as described above.

In addition to the above-described multiple-hop approach, the preferredembodiment of the present invention provides a particularly advantageousway of minimizing delay, cost and degradation of video quality in amulti-party video teleconference involving geographically dispersedsites, while still delivering full conference views of all participants.Normally, in order for the CMWs at all sites to be provided with liveaudio/video of every participant in a teleconference simultaneously,each site has to allocate (in router/codec bank 42 in FIG. 4) a separatecodec for each participant, as well as a like number of WAN trunks (viaWAN switching multiplexer 44 in FIG. 4).

As will next be described, however, the preferred embodiment of theinvention advantageously permits each wide area audio/videoteleconference to use only one codec at each site, and a minimum numberof WAN digital trunks. Basically, the preferred embodiment achieves thismost important result by employing “distributed” video mosaicing via avideo “cut-and-paste” technology along with distributed audio mixing.

Distributed Video Mosaicing

FIG. 7 illustrates a preferred way of providing video mosaicing in theMLAN of FIG. 3—i.e., by combining the individual analog video picturesfrom the individuals participating in a teleconference into a singleanalog mosaic picture. As shown in FIG. 7, analog video signals 112-1 to112-n from the participants of a teleconference are applied to videomosaicing circuitry 36, which in the preferred embodiment is provided aspart of conference bridge 35 in FIG. 3. These analog video inputs 112-1to 112-n are obtained from the A/V Switching Circuitry 30 (FIG. 3) andmay include video signals from CMWs at one or more distant sites(received via WAN gateway 40) as well as from other CMWs at the localsite.

In the preferred embodiment, video mosaicing circuitry 36 is capable ofreceiving N individual analog video picture signals (where N is asquared integer, i.e., 4, 9, 16, etc.). Circuitry 36 first reduces thesize of the N input video signals by reducing the resolutions of each bya factor of M (where M is the square root of N (i.e., 2, 3, 4, etc.),and then arranging them in an M-by-M mosaic of N images. The resultingsingle analog mosaic 36 a obtained from video mosaicing circuitry 36 isthen transmitted to the individual CMWs for display on the screensthereof.

As will become evident hereinafter, it may be preferable to send adifferent mosaic to distant sites, in which case video mosaicingcircuitry 36 would provide an additional mosaic 36 b for this purpose. Atypical displayed mosaic picture (N=4, M=2) showing three participantsis illustrated in FIG. 2A. A mosaic containing four participants isshown in FIG. 8B. It will be appreciated that, since a mosaic (36 a or36 b) can be transmitted as a single video picture to another site, viaWAN 15 (FIGS. 1 and 4), only one codec and digital trunk are required.Of course, if only a single individual video picture is required to besent from a site, it may be sent directly without being included in amosaic. Note that for large conferences it is possible to employmultiple video mosaics, one for each video window supported by the CMWs(see, e.g., FIG. 8C). In very large conferences, it is also possible todisplay video only from a select focus group whose members are selectedby a dynamic “floor control” mechanism. Also note that, with additionalmosaic hardware, it is possible to give each CMW its own mosaic. Thiscan be used in small conferences to raise the maximum number ofparticipants (from M sup 2 to M sup 2+1—i.e., 5, 10, 17, etc.) or togive everyone in a large conference their own “focus group” view.

Also note that the entire video mosaicing approach described thus farand continued below applies should digital video transmission be used inlieu of analog transmission, particularly since both mosaic and videowindow implementations use digital formats internally and in currentproducts are transformed to and from analog for external interfacing. Inparticular, note that mosaicing can be done digitally withoutdecompression with many existing compression schemes. Further, with anall-digital approach, mosaicing can be done as needed directly on theCMW.

FIG. 9 illustrates preferred audio mixing circuitry 38 for use inconjunction with the video mosaicing circuitry 36 in FIG. 7, both ofwhich may be part of conference bridges 35 in FIG. 3. As shown in FIG.9, audio signals 114-1 to 114-n are applied to audio summing circuitry38 for combination. These input audio signals 114-1 to 114-n may includeaudio signals from local participants as well as audio sums fromparticipants at distant sites. Audio mixing circuitry 38 provides arespective “minus-1” sum output 36 a-1, 36 a-2, etc., for eachparticipant. Thus, each participant hears every conference participant'saudio except his/her own.

In the preferred embodiment, sums are decomposed and formed in adistributed fashion, creating partial sums at one site which arecompleted at other sites by appropriate signal insertion. Accordingly,audio mixing circuitry 38 is able to provide one or more additionalsums, such as indicated by output 38, for sending to other sites havingconference participants.

Next to be considered is the manner in which video cut-and-pastetechniques are advantageously employed in the preferred embodiment. Itwill be understood that, since video mosaics and/or individual videopictures may be sent from one or more other sites, the problem arises asto how these situations are handled. In the preferred embodiment, videocut-and-paste circuitry 39, as illustrated in FIG. 10, is provided forthis purpose, and may also be incorporated in the conference bridges 35in FIG. 3.

Referring to FIG. 10, video cut-and-paste circuitry receives analogvideo inputs 116, which may be comprised of one or more mosaics orsingle video pictures received from one or more distant sites and amosaic or single video picture produced by the local site. It is assumedthat the local video mosaicing circuitry 36 (FIG. 7) and the videocut-and-paste circuitry 39 have the capability of handling all of theapplied individual video pictures, or at least are able to choose whichones are to be displayed based on existing available signals.

The video cut-and-paste circuitry 39 digitizes the incoming analog videoinputs 116, selectively rearranges the digital signals on aregion-by-region basis to produce a single digital M-by-M mosaic, havingindividual pictures in selected regions, and then converts the resultingdigital mosaic back to analog form to provide a single analog mosaicpicture 39 a for sending to local participants (and other sites whererequired) having the individual input video pictures in appropriateregions. This resulting cut-and-paste analog mosaic 39 a will providethe same type of display as illustrated in FIG. 8B. As will becomeevident hereinafter, it is sometimes beneficial to send differentcut-and-paste mosaics to different sites, in which case videocut-and-paste circuitry 39 will provide additional cut-and-paste mosaics39 b-1, 39 b-2, etc. for this purpose.

FIG. 11 diagrammatically illustrates an example of how videocut-and-paste circuitry may operate to provide the cut-and-paste analogmosaic 39 a. As shown in FIG. 11, four digitized individual signals 116a, 116 b, 116 c and 116 d derived from the input Video signals are“pasted” into selected regions of a digital frame buffer 17 to form adigital 2×2 mosaic, which is converted into an output analog videomosaic 39 a or 39 b in FIG. 10. The required audio partial sums may beprovided by audio mixing circuitry 39 in FIG. 9 in the same manner,replacing each cut-and-paste video operation with a partial sumoperation.

Having described in connection with FIGS. 7-11 how video mosaicing,audio mixing, video cut-and-pasting, and distributed audio mixing may beperformed, the following description of FIGS. 12-17 will illustrate howthese capabilities may advantageously be used in combination in thecontext of wide-area videoconferencing. For these examples, theteleconference is assumed to have four participants, designated as A, B,C and D, in which case 2×2 (quad) mosaics are employed. It is to beunderstood that greater numbers of participants could be provided. Also,two or more simultaneously occurring teleconferences could also behandled, in which case additional mosaicing, cut-and-paste and audiomixing circuitry would be provided at the various sites along withadditional WAN paths For each example, the “A” figure illustrates thevideo mosaicing and cut-and-pasting provided, and the corresponding “B”figure (having the same figure number) illustrates the associated audiomixing provided. Note that these figures indicate typical delays thatmight be encountered for each example (with a single “UNIT” delayranging from 0-450 milliseconds, depending upon available compressiontechnology).

FIGS. 12A and 12B illustrate a 2-site example having two participants Aand B at Site #1 and two participants C and D at Site #2 Note that thisexample requires mosaicing and cut-and-paste at both sites.

FIGS. 13A and 13B illustrate another 2-site example, but having threeparticipants A, B and C at Site #1 and one participant D at Site #2.Note that this example requires mosaicing at both sites, butcut-and-paste only at Site #2.

FIGS. 14A and 14B illustrate a 3-site example having participants A andB at Site #1, participant C at Site #2, and participant D at Site #3. AtSite #1, the two local videos A and B are put into a mosaic which issent to both Site #2 and Site #3. At Site #2 and Site #3, cut-and-pasteis used to insert the single video (C or D) at that site into the emptyregion in the imported A, B, and D or C mosaic, as shown. Accordingly,mosaicing is required at all three sites, and cut-and-paste is requiredfor only Site #2 and Site #3.

FIGS. 15A and 15B illustrate another 3-site example having participant Aat Site #1, participant B at Site #2, and participants C and D at Site#3. Note that mosaicing and cut-and-paste are required at all sites Site#2 additionally has the capability to send different cut-and-pastemosaics to Sites #1 and Site #3. Further note with respect to FIG. 15Bthat Site #2 creates minus-1 audio mixes for Site #1 and Site #2, butonly provides a partial audio mix (A&B) for Site #3. These partial mixesare completed at Site #3 by mixing in C's signal to complete D's mix(A+B+C) and D's signal to complete C's mix (A+B+D).

FIG. 16 illustrates a 4-site example employing a star topology, havingone participant at each site; that is, participant A is at Site #1,participant B is at Site #2, participant C is at Site #3, andparticipant D is at Site #4. An audio implementation is not illustratedfor this example, since standard minus-1 mixing can be performed at Site#1, and the appropriate sums transmitted to the other sites.

FIGS. 17A and 17B illustrate a 4-site example that also has only oneparticipant at each site, but uses a line topology rather than a startopology as in the example of FIG. 16. Note that this example requiresmosaicing and cut-and-paste at all sites. Also note that Site #2 andSite #3 are each required to transmit two different types ofcut-and-paste mosaics.

The preferred embodiment also provides the capability of allowing aconference participant to select a close-up of a participant displayedon a mosaic. This capability is provided whenever a full individualvideo picture is available at that user's site. In such case, the A/VSwitching Circuitry 30 (FIG. 3) switches the selected full video picture(whether obtained locally or from another site) to the CMW that requeststhe close-up.

Next to be described in connection with FIGS. 18A, 18B, 19 and 20 arevarious preferred embodiments of a CMW in accordance with the invention.

Collaborative Multimedia Workstation Hardware

One embodiment of a CMW 12 of the present invention is illustrated inFIG. 18A Currently available personal computers (e.g., an AppleMacintosh or an IBM-compatible PC, desktop or laptop) and workstations(e,g., a Sun SPARC station) can be adapted to work with the presentinvention to provide such features as real-time videoconferencing, dataconferencing, multimedia mail, etc. In business situations, it can beadvantageous to set up a laptop to operate with reduced functionalityvia cellular telephone links and removable storage media (e.g., CD-ROM,video tape with timecode support etc.), but take on full capability backin the office via a docking station connected to the MLAN 10. Thisrequires a voice and data modem as yet another function server attachedto the MLAN.

The currently available personal computers and workstations serve as abase workstation platform. The addition of certain audio and video I/Odevices to the standard components of the base platform 100 (wherestandard components include the display monitor 200, keyboard 300 andmouse or tablet (or other pointing device) 400), all of which connectwith the base platform box through standard peripheral ports 101, 102and 103, enables the CMW to generate and receive real-time audio andvideo signals. These devices include a video camera 500 for capturingthe user's image, gestures and surroundings (particularly the user'sface and upper body), a microphone 600 for capturing the user's spokenwords (and any other sounds generated at the CMW), a speaker 700 forpresenting incoming audio signals (such as the spoken words of anotherparticipant to a videoconference or audio annotations to a document), avideo input card 130 in the base platform 100 for capturing incomingvideo signals (e.g., the image of another participant to avideoconference, or videomail), and a video display card 120 fordisplaying video and graphical output on monitor 200 (where video istypically displayed in a separate window).

These peripheral audio and video I/O devices are readily available froma variety of vendors and are just beginning to become standard featuresin (and often physically integrated into the monitor and/or baseplatform of) certain personal computers and workstations. See, e.g., theaforementioned BYTE article (“Video Conquers the Desktop”), whichdescribes current models of Apple's Macintosh AV series personalcomputers and Silicon Graphics' Indy workstations.

Add-on box 800 (shown in FIG. 18A and illustrated in greater detail inFIG. 19) integrates these audio and video I/O devices with additionalfunctions (such as adaptive echo canceling and signal switching) andinterfaces with AV Network 901 Network 901 is the part of the MLAN 10which carries bidirectional audio and video signals among the CMWs andA/V Switching Circuitry 30—e.g., utilizing existing UTP wiring to carryaudio and video signals (digital or analog, as in the presentembodiment).

In the present embodiment, the AV network 901 is separate and distractfrom the Data Network 902 portion of the MLAN 10, which carriesbidirectional data signals among the CMWs and the Data LAN hub (e.g., anEthernet network that also utilizes UTP wiring in the present embodimentwith a network interface card 110 in each CMW). Note that each CMW willtypically be a node on both the AV and the Data Networks.

There are several approaches to implementing Add-on box 800. In atypical videoconference, video camera 500 and microphone 600 capture andtransmit outgoing video and audio signals into ports 801 and 802,respectively, of Add-on box 800. These signals are transmitted viaAudio/Video I/O port 805 across AV Network 901. Incoming video and audiosignals (from another videoconference participant) are received acrossAV network 901 through Audio/Video 110 port 805. The video signals aresent out of V-OUT port 803 of CMW add-on box 800 to video input card 130of base platform 100, where they are displayed (typically in a separatevideo window) on monitor 200 utilizing the standard base platform videodisplay card 120. The audio signals are sent out of A-OUT port 804 ofCMW add-on box 800 and played through speaker 700 while the videosignals are displayed on monitor 200. The same signal flow occurs forother non-teleconferencing applications of audio and video.

Add-on box 800 can be controlled by CMW software (illustrated in FIG.20) executed by base platform 100 Control signals can be communicatedbetween base platform port 104 and Add-on box Control port 806 (e.g., anRS-232, Centronics, SCSI or other standard communications port).

Many other embodiments of the CMW illustrated in FIG. 18A will work inaccordance with the present invention. For example, Add-on box 800itself can be implemented as an add-in card to the base platform 100Connections to the audio and video I/O devices need not change, thoughthe connection for base platform control can be implemented internally(e.g., via the system bus) rather than through an external RS-232 orSCSI peripheral port. Various additional levels of integration can alsobe achieved as will be evident to those skilled in the art. For example,microphones, speakers, video cameras and UTP transceivers can beintegrated into the base platform 100 itself, and all media handlingtechnology and communications can be integrated onto a single card.

A handset/headset jack enables the use of an integrated audio I/O deviceas an alternate to the separate microphone and speaker. A telephoneinterface could be integrated into add-on box 800 as a localimplementation of computer-integrated telephony. A “hold” (i.e., audioand video mute) switch and/or a separate audio mute switch could beadded to Add-on box 800 if such an implementation were deemed preferableto a software-based interface.

The internals of Add-on box 800 of FIG. 18A are illustrated in FIG. 19.Video signals generated at the CMW (e.g., captured by camera 500 of FIG.18A) are sent to CMW add-on box 800 via V-IN port 801. They thentypically pass unaffected through Loopback/AV Mute circuitry 830 viavideo ports 833 (input) and 834 (output) and into A/V Transceivers 840(via Video In port 842) where they are transformed from standard videocable signals to UTP signals and sent out via port 845 and Audio/VideoI/O port 805 onto AV Network 901.

The Loopback/AV Mute circuitry 830 can, however, be placed in variousmodes under software control via Control port 806 (implemented, forexample, as a standard UART). If in loopback mode (e.g., for testingincoming and outgoing signals at the CMW), the video signals would berouted back out V-OUT port 803 via video port 831. If in a mute mode(e.g., muting audio, video or both), video signals might, for example,be disconnected and no video signal would be sent out video port 834Loop back and muting switching functionality is also provided for audioin a similar way. Note: that computer control of loopback is very usefulfor remote testing and diagnostics while manual override of computercontrol on mute is effective for assured privacy from use of theworkstation for electronic spying.

Video input (e.g., captured by the video camera at the CMW of anothervideoconference participant) is handled in a similar fashion. It isreceived along AV Network 901 through Audio/Video I/O port 805 and port845 of A/V Transceivers 840, where it is sent out Video Out port 841 tovideo port 832 of Loopback/AV Mute circuitry 830, which typically passessuch signals out video port 831 to V-OUT port 803 (for receipt by avideo input card or other display mechanism, such as LCD display 810 ofCMW Side Mount unit 850 in FIG. 18B, to be discussed).

Audio input and output (e.g., for playback through speaker 700 andcapture by microphone 600 of FIG. 18A) passes through A/V transceivers840 (via Audio In port 844 and Audio Out port 843) and Loopback/AV Mutecircuitry 830 (through audio ports 837/838 and 836/835) in a similarmanner. The audio input and output ports of Add-on box 800 interfacewith standard amplifier and equalization circuitry, as well as anadaptive room echo canceler 814 to eliminate echo, minimize feedback andprovide enhanced audio performance when using a separate microphone andspeaker. In particular, use of adaptive room echo cancelers provideshigh-quality audio interactions in wide area conferences. Becauseadaptive room echo canceling requires training periods (typicallyinvolving an objectionable blast of high-amplitude white noise or tonesequences) for alignment with each acoustic environment, it is preferredthat separate echo canceling be dedicated to each workstation ratherthan sharing a smaller group of echo cancelers across a larger group ofworkstations.

Audio inputs passing through audio port 835 of Loopback/AV Mutecircuitry 830 provide audio signals to a speaker (via standard EchoCanceler circuitry 814 and A-OUT port 804 and/or to a handset or headset(via I/O ports 807 and 808, respectively, under volume control circuitry815 controlled by software through Control port 806). In all cases,incoming audio signals pass through power amplifier circuitry 812 beforebeing sent out of Add-on box 80) to the appropriate audio-emittingtransducer.

Outgoing audio signals generated at the CMW (e.g., by microphone 600 ofFIG. 18A or the mouthpiece of a handset or headset) enter Add-on box 800via A-IN port 802 (for a microphone) or Handset or Headset I/O ports 807and 808, respectively. In all cases, outgoing audio signals pass throughstandard preamplifier (811) and equalization (813) circuitry, whereuponthe desired signal is selected by standard “Select” switching circuitry816 (under software control through Control port 806) and passed toaudio port 837 of Loopback/AV Mute circuitry, 830.

It is to be understood that A/V Transceivers 840 may includemuxing/demuxing facilities so as to enable the transmission ofaudio/video signals on a single pair of wires, e.g., by encoding audiosignals digitally in the vertical retrace interval of the analog videosignal. Implementation of other audio and video enhancements, such asstereo audio and external audio/video I/O ports (e.g., for recordingsignals generated at the CMW), are also well within the capabilities ofone skilled in the art. If stereo audio is used in teleconferencing(i.e., to create useful spatial metaphors for users), a second echocanceller may be recommended.

Another embodiment of the CMW of this invention, illustrated in FIG.18B, utilizes a separate (fully self-contained) “Side Mount” approachwhich includes its own dedicated video display. This embodiment isadvantageous in a variety of situations, such as instances in whichadditional screen display area is desired (e.g., in a laptop computer ordesktop system with a small monitor) or where it is impossible orundesirable to retrofit older, existing or specialized desktop computersfor audio/video support. In this embodiment, video camera 500,microphone 600 and speaker 700 of FIG. 18A are integrated together withthe functionality of Add-on box 80). Side Mount 850 eliminates thenecessity of external connections to these integrated audio and videoI/O devices, and includes an LCD display 810 for displaying the incomingvideo signal (which thus eliminates the need for a base platform videoinput card 130).

Given the proximity of Side Mount device 850 to the user, and the directaccess to audio/video I/O within that device, various additionalcontrols 820 can be provided at the user's touch (all well within thecapabilities of those skilled in the art). Note that, with enoughadditions, Side Mount unit 850 can become virtually a standalone devicethat does not require a separate computer for services using only audioand video. This also provides a way of supplementing a network offull-feature workstations with a few low-cost additional “audio videointercoms” for certain sectors of an enterprise (such as clerical,reception, factory floor, etc.).

A portable laptop implementation can be made to deliver multimedia mailwith video, audio and synchronized annotations via CD-ROM or an add-onvideotape unit with separate video, audio and time code tracks (a stereovideotape player can use the second audio channel for time codesignals). Videotapes or CD-ROMs can be created in main offices andexpress mailed, thus avoiding the need for high-bandwidth networkingwhen on the road. Cellular phone links can be used to obtain both voiceand data communications (via modems) Modem-based data communications aresufficient to support remote control of mail or presentation playback,annotation, file transfer and fax features. The laptop can then bebrought into the office and attached to a docking station where theavailable MLAN 10 and additional functions adapted from Add-on box 800can be supplied, providing full CMW capability.

Collaborative Multimedia Workstation Software

CMW software modules 160 are illustrated generally in FIG. 20 anddiscussed in greater detail below in conjunction with the softwarerunning on MLAN Server 60 of FIG. 3 Software 160 allows the user toinitiate and manage (in conjunction with the server software)videoconferencing, data conferencing, multimedia mail and othercollaborative sessions with other users across the network.

Also present on the CMW in this embodiment are standard multitaskingoperating system/GUI software 180 (e.g., Apple Macintosh System 7,Microsoft Windows 3.1, or UNIX with the “X Window System” and Motif orother GUI “window manager” software) as well as other applications 170,such as word processing and spreadsheet programs Software modules161-168 communicate with operating system/GUI software 180 and otherapplications 170 utilizing standard function calls and interapplicationprotocols.

The central component of the Collaborative Multimedia Workstationsoftware is the Collaboration Initiator 161. All collaborative functionscan be accessed through this module When the Collaboration Initiator isstarted, it exchanges initial configuration information with the AudioVideo Network Manager (AVNM) 60 (shown in FIG. 3) through Data Network902. Information is also sent from the Collaboration Initiator to theAVNM indicating the location of the user, the types of servicesavailable on that workstation (e.g., videoconferencing, dataconferencing, telephony, etc.) and other relevant initializationinformation.

The Collaboration Initiator presents a user interface that allows theuser to initiate collaborative sessions (both real-time andasynchronous). In the preferred embodiment, session participants can beselected from a graphical rolodex 163 that contains a scrollable list ofuser names or from a list of quick-dial buttons 162 Quick-dial buttonsshow the face icons for the users they represent. In the preferredembodiment, the icon representing the user is retrieved by theCollaboration Initiator from the Directory Server 66 on MLAN Server 60when it starts up Users can dynamically add new quick-dial buttons bydragging the corresponding entries from the graphical rolodex onto thequick-dial panel.

Once the user elects to initiate a collaborative session, he or sheselects one or more desired participants by, for example, clicking onthat name to select the desired participant from the system rolodex or apersonal rolodex, or by clicking on the quick-dial button for thatparticipant (see, e.g., FIG. 2A). In either case, the user then selectsthe desired session type—e.g., by clicking on a CALL button to initiatea videoconference call, a SHARE button to initiate the sharing of asnapshot image or blank whiteboard, or a MAIL button to send mail.Alternatively, the user can double-click on the rolodex name or a faceicon to initiate the default session type—e.g., an audio/videoconference call.

The system also allows sessions to be invoked from the keyboard. Itprovides a graphical editor to bind combinations of participants andsession types to certain hot keys. Pressing this hot key (possibly inconjunction with a modifier key, e.g., <Shift> or <Ctrl>) will cause theCollaboration Initiator to start a session of the specified type withthe given participants.

Once the user selects the desired participant and session type,Collaboration Initiator module 161 retrieves necessary addressinginformation from Directory Service 66 (see FIG. 21). In the case of avideoconference call, the Collaboration Initiator then communicates withthe AVNM (as described in greater detail below) to set up the necessarydata structures and manage the various states of that call, and tocontrol A/V Switching Circuitry 30, which selects the appropriate audioand video signals to be transmitted to/from each participant's CMW. Inthe case of a data conferencing session, the Collaboration Initiatorlocates, via the AVNM, the Collaboration Initiator modules at the CMWsof the chosen recipients, and sends a message causing the CollaborationInitiator modules to invoke the Snapshot Sharing modules 164 at eachparticipant's CMW. Subsequent videoconferencing and data conferencingfunctionality is discussed in greater detail below in the context ofparticular usage scenarios.

As indicated previously, additional collaborative services—such as Mail165, Application Sharing 166, Computer-Integrated Telephony 167 andComputer Integrated Fax 168—are also available from the CMW by utilizingCollaboration Initiator module 161 to initiate the session (i.e., tocontact the participants) and to invoke the appropriate applicationnecessary to manage the collaborative session When initiatingasynchronous collaboration (e.g., mail, fax, etc.), the CollaborationInitiator contacts Directory Service 66 for address information (e.g.,EMAIL address, fax number, etc.) for the selected participants andinvokes the appropriate collaboration tools with the obtained addressinformation. For real-time sessions, the Collaboration Initiator queriesthe Service Server module 69 inside AVNM 63 for the current location ofthe specified participants. Using this location information, itcommunicates (via the AVNM) with the Collaboration Initiators of theother session participants to coordinate session setup. As a result thevarious Collaboration Initiators will invoke modules 166, 167 or 168(including activating any necessary devices such as the connectionbetween the telephone and the CMW's audio I/O port). Further details onmultimedia mail are provided below.

MLAN Server Software

FIG. 21 diagrammatically illustrates software 62 comprised of variousmodules (as discussed above) provided for running on MLAN Server 60(FIG. 3) in the preferred embodiment. It is to be understood thatadditional software modules could also be provided. It is also to beunderstood that, although the software illustrated in FIG. 21 offersvarious significant advantages, as will become evident hereinafter,different forms and arrangements of software may also be employed withinthe scope of the invention. The software can also be implemented invarious sub-parts running as separate processes.

In the preferred embodiment, clients (e.g., software-controllingworkstations, VCRs, laserdisks, multimedia resources, etc.) communicatewith the MLAN Server Software Modules 62 using the TCP/IP networkprotocols. Generally, the AVNM 63 cooperates with the Service Server 69,Conference Bridge Manager (CBM 64 in FIG. 21) and the WAN NetworkManager (WNM 65 in FIG. 21) to manage communications within and amongboth MLANs 10 and WANs 15 (FIGS. 1 and 3).

The AVNM additionally cooperates with Audio/Video Storage Server 67 andother multimedia services 68 in FIG. 21 to support various types ofcollaborative interactions as described herein CBM 64 in FIG. 21operates as a client of the AVNM 63 to manage conferencing bycontrolling the operation of conference bridges 35. This includesmanagement of the video mosaicing circuitry 37, audio mixing circuitry38 and cut-and-paste circuitry 39 preferably incorporated therein WNM 65manages the allocation of paths (codecs and trunks) provided by WANgateway 40 for accomplishing the communications to other sites calledfor by the AVNM.

Audio Video Network Manager

The AVNM 63 manages A/V Switching Circuitry 30 in FIG. 3 for selectivelyrouting audio/video signals to and from CMWs 12, and also to and fromWAN gateway 40, as called for by clients Audio/video devices (e.g., CMWs12, conference bridges 35, multimedia resources 16 and WAN gateway 40 inFIG. 3) connected to A/V Switching Circuitry 30 in FIG. 3, have physicalconnections for audio in, audio out, video in and video out. For eachdevice on the network, the AVNM combines these four connections into aport abstraction, wherein each port represents an addressablebidirectional audio/video channel. Each device connected to the networkhas at least one port Different ports may share the same physicalconnections on the switch. For example, a conference bridge maytypically have four ports (for 2×2 mosaicing) that share the samevideo-out connection. Not all devices need both video and audioconnections at a port. For example, a TV tuner port needs only incomingaudio/video connections.

In response to client program requests, the AVNM provides connectivitybetween audio/video devices by connecting their ports Connecting portsis achieved by switching one port's physical input connections to theother port's physical output connections (for both audio and video) andvice-versa Client programs can specify which of the 4 physicalconnections on its ports should be switched. This allows client programsto establish unidirectional calls (e.g., by specifying that only theport's input connections should be switched and not the port's outputconnections) and audio-only or video-only calls (by specifying audioconnections only or video connections only).

Service Server

Before client programs can access audio/video resources through theAVNM, they must register the collaborative services they provide withthe Service Server 69. Examples of these services indicate “video call”,“snapshot sharing”, “conference” and “video file sharing.” These servicerecords are entered into the Service Server's service database. Theservice database thus keeps track of the location of client programs andthe types of collaborative sessions in which they can participate. Thisallows the Collaboration Initiator to find collaboration participants nomatter where they are located. The service database is replicated by allService Servers: Service Servers communicate with other Service Serversin other MLANs throughout the system to exchange their service records.

Clients may create a plurality of services, depending on thecollaborative capabilities desired. When creating a service, a clientcan specify the network resources (e.g ports) that will be used by thisservice. In particular, service information is used to associate a userwith the audio/video ports physically connected to the particular CMWinto which the user is logged in Clients that want to receive requestsdo so by putting their services in listening mode. If clients want toaccept incoming data shares, but want to block incoming video calls,they must create different services.

A client can create an exclusive service on a set of ports to preventother clients from creating services on these ports. This is useful, forexample, to prevent multiple conference bridges from managing the sameset of conference bridge ports.,

Next to be considered is the preferred manner in which the AVNM 63 (FIG.21), in cooperation with the Service Server 69, CBM 64 and participatingCMWs provide for managing. A/V Switching Circuitry 30 and conferencebridges 35 in FIG. 3 during audio/video/data teleconferencing. Theparticipating CMWs may include workstations located at both local andremote sites.

Basic Two-Party Videoconferencing

As previously described, a CMW includes a Collaboration Initiatorsoftware module 161 (see FIG. 20) which is used to establishperson-to-person and multiparty calls. The corresponding collaborationinitiator window advantageously provides quick-dial face icons offrequently dialed persons, as illustrated, for example, in FIG. 22,which is an enlarged view of typical face icons along with variousinitiating buttons (described in greater detail below in connection withFIGS. 35-42).

Videoconference calls can be initiated, for example, merely bydouble-clicking on these icons. When a call is initiated, the CMWtypically provides a screen display that includes a live video pictureof the remote conference participant, as illustrated for example in FIG.8A. In the preferred embodiment, this display also includes controlbuttons/menu items that can be used to place the remote participant onhold, to resume a call on hold, to add one or more participants to thecall, to initiate data sharing and to hang up the call.

The basic underlying software-controlled operations occurring for atwo-party call are diagrammatically illustrated in FIG. 23 When a callerinitiates a call (e.g., by selecting a user from the graphical rolodexand clicking the call button or by double-clicking the face icon of thecallee on the quick-dial panel), the caller's Collaboration Initiatorresponds by identifying the selected user and requesting that user'saddress from Directory Service 66, as indicated by (2) in FIG. 23Directory Service 66 looks up the callee's address in the directorydatabase, as indicated by (3) in FIG. 23, and then returns it to thecaller's Collaboration Initiator, as illustrated by (4) in FIG. 23.

The caller's Collaboration Initiator sends a request to the AVNM toplace a video call to caller with the specified address, as indicated by(5) in FIG. 23. The AVNM queries the Service Server to find the serviceinstance of type “video call” whose name corresponds to the callee'saddress. This service record identifies the location of the callee'sCollaboration Initiator as well as the network ports that the callee isconnected to. If no service instance is found for the callee, the AVNMnotifies the caller that the callee is not logged in. If the callee islocal, the AVNM sends a call event to the callee's CollaborationInitiator, as indicated by (6) in FIG. 23. If the callee is at a remotesite, the AVNM forwards the call request (5) through the WAN gateway 40for transmission, via WAN 15 (FIG. 1) to the Collaboration Initiator ofthe callee's CMW at the remote site.

The callee's Collaboration Initiator can respond to the call event in avariety of ways. In the preferred embodiment, a user-selectable sound isgenerated to announce the incoming call. The Collaboration Initiator canthen act in one of two modes. In “Telephone Mode,” the CollaborationInitiator displays an invitation message on the CMW screen that containsthe name of the caller and buttons to accept or refuse the call. TheCollaboration Initiator will then accept or refuse the call, dependingon which button is pressed by the callee In “Intercom Mode,” theCollaboration Initiator accepts all incoming calls automatically, unlessthere is already another call active on the callee's CMW, in which casebehavior reverts to Telephone Mode.

The callee's Collaboration Initiator then notifies the AVNM as towhether the call will be accepted or refused. If the call is accepted,(7), the AVNM sets up the necessary communication paths between thecaller and the callee required to establish the call. The AVNM thennotifies the caller's Collaboration Initiator that the call has beenestablished by sending it an accept event (8). If the caller and calleeare at different sites, their AVNMs will coordinate in setting up thecommunication paths at both sites, as required by the call.

The AVNM may provide for managing connections among CMWs and othermultimedia resources for audio/video/data communications in variousways. The manner employed in the preferred embodiment will next bedescribed.

As has been described previously, the AVNM manages the switches in theA/V Switching Circuitry 30 in FIG. 3 to provide port-to-port connectionsin response to connection requests from clients. The primary datastructure used by the AVNM for managing these connections will bereferred to as a callhandle, which is comprised of a plurality of bits,including state bits.

Each port-to-port connection managed by the AVNM comprises twocallhandles, one associated with each end of the connection. Thecallhandle at the client port of the connection permits the client tomanage the client's end of the connection. The callhandle mode bitsdetermine the current state of the callhandle and which of a port's fourswitch connections (video in, video out, audio in, audio out) areinvolved in a call.

AVNM clients send call requests to the AVNM whenever they want toinitiate a call. As part of a call request, the client specifies thelocal service in which the call will be involved, the name of thespecific port to use for the call; identifying information as to thecallee, and the call mode. In response, the AVNM creates a callhandle onthe caller's port.

All callhandles are created in the “idle” state. The AVNM then puts thecaller's callhandle in the “active” state. The AVNM next creates acallhandle for the callee and sends it a call event, which places thecallee's callhandle in the “ringing” state. When the callee accepts thecall, its callhandle is placed in the “active” state, which results in aphysical connection between the caller and the callee. Each port canhave an arbitrary number of callhandles bound to it, but typically onlyone of these callhandles can be active at the same time.

After a call has been set up, AVNM clients can send requests to the AVNMto change the state of the call, which can advantageously beaccomplished by controlling the callhandle states. For example, during acall, a call request from another party could arrive. This arrival couldbe signaled to the user by providing an alert indication in a dialoguebox on the user's CMW screen. The user could refuse the call by clickingon a refuse button in the dialogue box, or by clicking on a “hold”button on the active call window to put the current call on hold andallow the incoming call to be accepted.

The placing of the currently active call on hold can advantageously beaccomplished by changing the caller's callhandle from the active stateto a “hold” state, which permits the caller to answer incoming calls orinitiate new calls, without releasing the previous call. Since theconnection set-up to the callee will be retained, a call on hold canconveniently be resumed by the caller clicking on a resume button on theactive call window, which returns the corresponding callhandle back tothe active state. Typically, multiple calls can be put on hold in thismanner. As an aid in managing calls that are on hold, the CMWadvantageously provides a hold list display, identifying these on-holdcalls and (optionally) the length of time that each party is on hold. Acorresponding face icon could be used to identify each on-hold call. Inaddition, buttons could be provided in this hold display which wouldallow the user to send a preprogrammed message to a party on hold. Forexample, this message could advise the callee when the call will beresumed, or could state that the call is being terminated and will bereinitiated at a later time.

Reference is now directed to FIG. 24 which diagrammatically illustrateshow two-party calls are connected for CMWs WS-1 and WS-2, located at thesame MLAN 10. As shown in FIG. 24, CMWs WS-1 and WS-2 are coupled to thelocal A/V Switching Circuitry 30 via ports 81 and 82, respectively. Aspreviously described, when CMW WS-1 calls CMW WS-2, a callhandle iscreated for each port. If CMW WS-2 accepts the call, these twocallhandles become active and in response thereto, the AVNM causes theA/V Switching Circuitry 30 to set up the appropriate connections betweenports 81 and 82, as indicated by the dashed line 83.

FIG. 25 diagrammatically illustrates how two-party calls are connectedfor CMWs WS-1 and WS-2 when located in different MLANs 10 a and 10 b. Asillustrated in FIG. 25, CMW WS-1 of MLAN 10 a is connected to a port 91a of A/V Switching Circuitry 30 a of MLAN 10 a, while CMW WS-2 isconnected to a port 91 b of the A/V Switching Circuitry 30 b of MLAN 10b. It will be assumed that MLANs 10 a and 10 b can communicate with eachother via ports 92 a and 92 b (through respective WAN gateways 40 a and40 b and WAN 15). A call between CMWs WS-1 and WS-2 can then beestablished by AVNM of MLAN 10 a in response to the creation ofcallhandles at ports 91 a and 92 a, setting up appropriate connectionsbetween these ports as indicated by dashed line 93 a, and by AVNM ofMLAN 10 b, in response to callhandles created at ports 91 b and 92 b,setting up appropriate connections between these ports as indicated bydashed line 93 b. Appropriate paths 94 a and 94 b in WAN gateways 40 aand 40 b, respectively, are set up by the WAN network manager 65 (FIG.21) in each network.

Conference Calls

Next to be described is the specific manner in which the preferredembodiment provides for multi-party conference calls (involving morethan two participants) When a multi-party conference call is initiated,the CMW provides a screen that is similar to the screen for two-partycalls, which displays a live video picture of the callee's image in avideo window. However, for multi-party calls, the screen includes avideo mosaic containing a live video picture of each of the conferenceparticipants (including the CMW user's own picture), as shown, forexample, in FIG. 8B. Of course, other embodiments could show only theremote conference participants (and not the local CMW user) in theconference mosaic (or show a mosaic containing both participants in atwo-party call). In addition to the controls shown in FIG. 8B, themulti-party conference screen also includes buttons/menu items that canbe used to place individual conference participants on hold, to removeindividual participants from the conference, to adjourn the entireconference, or to provide a “close-up” image of a single individual (inplace of the video mosaic).

Multi-party conferencing requires all the mechanisms employed for2-party calls. In addition, it requires the conference bridge managerCBM 64 (FIG. 21) and the conference bridges 36 (FIG. 3). The CBM acts asa client of the AVNM in managing the operation of the conference bridges36. The CBM also acts a server to other clients on the network. The CBMmakes conferencing services available by creating service records oftype “conference” in the AVNM service database and associating theseservices with the ports on A/V Switching Circuitry 30 for connection toconference bridges 36.

The preferred embodiment provides two ways for initiating a conferencecall. The first way is to add one or more parties to an existingtwo-party call. For this purpose, an ADD button is provided by both theCollaboration Initiator and the Rolodex, as illustrated in FIGS. 2A and22. To add a new party, a user selects the party to be added (byclicking on the user's rolodex name or face icon as described above) andclicks on the ADD button to invite that new party. Additional partiescan be invited in a similar manner. The second way to initiate aconference call is to select the parties in a similar manner and thenclick on the CALL button (also provided in the Collaboration Initiatorand Rolodex windows on the user's CMW screen).

Another alternative embodiment is to initiate a conference call from thebeginning by clicking on a CONFERENCE/MOSAIC icon/button/menu item onthe CMW screen. This could initiate a conference call with the callinitiator as the sole participant (i.e., causing a conference bridge tobe allocated such that the caller's image also appears on his/her ownscreen in a video mosaic, which will also include images of subsequentlyadded participants). New participants could be invited, for example, byselecting each new party's face icon and then clicking on the ADDbutton.

Next to be considered with reference to FIGS. 26 and 27 is the manner inwhich conference calls are handled in the preferred embodiment. For thepurposes of this description it will be assumed that up to four partiesmay participate in a conference call. Each conference uses four bridgeports 136-1, 136-2, 136-3 and 136-4 provided on A/V Switching Circuitry30 a, which are respectively coupled to bidirectional audio/video lines36-1, 36-2, 36-3 and 36-4 connected to conference bridge 36. However,from this description it will be apparent how a conference call may beprovided for additional parties, as well as simultaneously occurringconference calls.

Once the Collaboration Initiator determines that a conference is to beinitiated, it queries the AVNM for a conference service. If such aservice is available, the Collaboration Initiator requests theassociated CBM to allocate a conference bridge. The CollaborationInitiator then places an audio/video call to the CBM to initiate theconference When the CBM accepts the call, the AVNM couples port 101 ofCMW WS-1 to lines 36-1 of conference bridge 36 by a connection 137produced in response to callhandles created for port 101 of WS-1 andbridge port 136-1.

When the user of WS-1 selects the appropriate face icon and clicks theADD button to invite a new participant to the conference, which will beassumed to be CMW WS-3, the Collaboration initiator on WS-1 sends an addrequest to the CBM. In response, the CBM calls WS-3 via WS-3 port 103When CBM initiates the call, the AVNM creates callhandles for WS-3 port103 and bridge port 136-2 When WS-3 accepts the call, its callhandle ismade “active,” resulting in connection 138 being provided to connectWS-3 and lines 136-2 of conference bridge 36. Assuming CMW WS-1 nextadds CMW WS-5 and then CMW WS-8, callhandles for their respective portsand bridge ports 136-3 and 136-4 are created, in turn, as describedabove for WS-1 and WS-3, resulting in connections 139 and 140 beingprovided to connect WS-5 and WS-9 to conference bridge lines 36-3 and36-4, respectively. The conferees WS-1, WS-3, WS-5 and WS-8 are thuscoupled to conference bridge lines 136-1, 136-2, 136-3 and 136-4,respectively as shown in FIG. 26.

It will be understood that the video mosaicing circuitry 36 and audiomixing circuitry 38 incorporated in conference bridge 36 operate aspreviously described, to form a resulting four-picture mosaic (FIG. 8B)that is sent to all of the conference participants, which in thisexample are CMWs WS-1, WS-2, WS-5 and WS-8 Users may leave a conferenceby just hanging up, which causes the AVNM to delete the associatedcallhandles and to send a hangup notification to CBM When CBM receivesthe notification, it notifies all other conference participants that theparticipant has exited. In the preferred embodiment, this results in ablackened portion of that participant's video mosaic image beingdisplayed on the screen of all remaining participants.

The manner in which the CBM and the conference bridge 36 operate whenconference participants are located at different sites will be evidentfrom the previously described operation of the cut-and-paste circuitry39 (FIG. 10) with the video mosaicing circuitry 36 (FIG. 7) and audiomixing circuitry 38 (FIG. 9). In such case, each incoming single videopicture or mosaic from another site is connected to a respective one ofthe conference bridge lines 36-1 to 36-4 via WAN gateway 40.

The situation in which a two-party call is converted to a conferencecall will next be considered in connection with FIG. 27 and thepreviously considered 2-party call illustrated in FIG. 24 Convertingthis 2-party call to a conference requires that this two-party call(such as illustrated between WS-1 and WS-2 in FIG. 24) be rerouteddynamically so as to be coupled through conference bridge 36 When theuser of WS-1 clicks on the ADD button to add a new party (for exampleWS-5), the Collaboration Initiator of WS-1 sends a redirect request tothe AVNM, which cooperates with the CBM to break the two-partyconnection 83 in FIG. 24, and then redirect the callhandles created forports 81 and 83 to callhandles created for bridge ports 136-1 and 136-2,respectively.

As shown in FIG. 27, this results in producing a connection 86 betweenWS-1 and bridge port 136-1, and a connection 87 between WS-2 and bridgeport 136-2, thereby creating a conference set-up between WS-1 and WS-2.Additional conference participants can then be added as described abovefor the situations described above in which the conference is initiatedby the user of WS-1 either selecting multiple participants initially ormerely selecting a “conference” and then adding subsequent participants.

Having described the preferred manner in which two-party calls andconference calls are set up in the preferred embodiment, the preferredmanner in which data conferencing is provided between CMWs will next bedescribed.

Data Conferencing

Data conferencing is implemented in the preferred embodiment by certainSnapshot Sharing software provided at the CMW (see FIG. 20). Thissoftware permits a “snapshot” of a selected portion of a participant'sCMW screen (such as a window) to be displayed on the CMW screens ofother selected participants (whether or not those participants are alsoinvolved in a videoconference). Any number of snapshots may be sharedsimultaneously. Once displayed, any participant can then telepoint on orannotate the snapshot, which animated actions and results will appear(virtually simultaneously) on the screens of all other participants. Theannotation capabilities provided include lines of several differentwidths and text of several different sizes. Also, to facilitateparticipant identification, these annotations may be provided in adifferent color for each participant. Any annotation may also be erasedby any participant FIG. 2B (lower left window) illustrates a CMW screenhaving a shared graph on which participants have drawn and typed to callattention to or supplement specific portions of the shared image.

A participant may initiate data conferencing with selected participants(selected and added as described above for videoconference calls) byclicking on a SHARE button on the screen (available in the Rolodex orCollaboration Initiator windows, shown in FIG. 2A, as are CALL and ADDbuttons), followed by selection of the window to be shared When aparticipant clicks on his SHARE button, his Collaboration Initiatormodule 161 (FIG. 20) queries the AVNM to locate the CollaborationInitiators of the selected participants, resulting in invocation oftheir respective Snapshot Sharing modules 164. The Snapshot Sharingsoftware modules at the CMWs of each of the selected participants querytheir local operating system 180 to determine available graphic formats,and then send this information to the initiating Snapshot Sharingmodule, which determines the format that will produce the mostadvantageous display quality and performance for each selectedparticipant

After the snapshot to be shared is displayed on all CMWs, eachparticipant may telepoint on or annotate the snapshot, which actions andresults are displayed on the CMW screens of all participants. This ispreferably accomplished by monitoring the actions made at the CMW (e.g.,by tracking mouse movements) and sending these “operating systemcommands” to the CMWs of the other participants, rather thancontinuously exchanging bitmaps, as would be the case with traditional“remote control” products.

As illustrated in FIG. 28, the original unchanged snapshot is stored ina first bitmap 210 a. A second bitmap 210 b stores the combination ofthe original snapshot and any annotations. Thus, when desired (e.g., byclicking on a CLEAR button located in each participant's Share window,as illustrated in FIG. 2B), the original unchanged snapshot can berestored (i.e., erasing all annotations) using bitmap 210 a Selectiveerasures can be accomplished by copying into (i.e., restoring) thedesired erased area of bitmap 210 b with the corresponding portion frombitmap 210 a.

Rather than causing a new Share window to be created whenever a snapshotis shared, it is possible to replace the contents of an existing Sharewindow with a new image. This can be achieved in either of two ways.First, the user can click on the GRAB button and then select a newwindow whose contents should replace the contents of the existing Sharewindow. Second, the user can click on the REGRAB button to cause a(presumably modified) version of the original source window to replacethe contents of the existing Share window. This is particularly usefulwhen one participant desires to share a long document that cannot bedisplayed on the screen in its entirety. or example, the user mightdisplay the first page of a spreadsheet on his screen, use the SHAREbutton to share that page, discuss and perhaps annotate it, then returnto the spreadsheet application to position to the next page, use theREGRAB button to share the new page, and so on. This mechanismrepresents a simple, effective step toward application sharing.

Further, instead of sharing a snapshot of data on his current screen, auser may instead choose to share a snapshot that had previously beensaved as a file. This is achieved via the LOAD button, which causes adialogue box to appear, prompting the user to select a file Conversely,via the SAVE button, any snapshot may be saved, with all currentannotations.

The capabilities described above were carefully selected to beparticularly effective in environments where the principal goal is toshare existing information, rather than to create new information. Inparticular, user interfaces are designed to make snapshot capture,telepointing and annotation extremely easy to use. Nevertheless, it isalso to be understood that, instead of sharing snapshots, a blank“whiteboard” can also be shared (via the WHITEBOARD button provided bythe Rolodex, Collaboration Initiator, and active call windows), and thatmore complex paintbox capabilities could easily be added for applicationareas that require such capabilities.

As pointed out previously herein, important features of the presentinvention reside in the manner in which the capabilities and advantagesof multimedia mail (MMM), multimedia conference recording (MMCR), andmultimedia document management (MMDM) are tightly integrated withaudio/video/data teleconferencing to provide a multimedia collaborationsystem that facilitates an unusually higher level of communication andcollaboration between geographically dispersed users than has heretoforebeen achievable by known prior an systems FIG. 29 is a schematic anddiagrammatic view illustrating how multimedia calls/conferences, MMCR,MMM and MMDM work together to provide the above-described features. Inthe preferred embodiment MM Editing Utilities shown supplementing MMMand MMDM may be identical.

Having already described various preferred embodiments and examples ofaudio/video/data teleconferencing, next to be considered are variouspreferred ways of integrating MMCR, MMM and MMDM with audio/video/datateleconferencing in accordance with the invention, or this purpose,basic preferred approaches and features of each will be considered alongwith preferred associated hardware and software.

Multimedia Documents

In the preferred embodiment, the creation, storage, retrieval andediting of multimedia documents serve as the basic element common toMMCR, MMM and MMDM. Accordingly, the preferred embodiment advantageouslyprovides a universal format for multimedia documents. This formatdefines multimedia documents as a collection of individual components inmultiple media combined with an overall structure and timing componentthat captures the identities, detailed dependencies, references to, andrelationships among the various other components. The informationprovided by this structuring component forms the basis for spatiallayout, order of presentation, hyperlinks, temporal synchronization,etc., with respect to the composition of a multimedia document FIG. 30shows the structure of such documents as well as their relationship withediting and storage facilities.

Each of the components of a multimedia document uses its own editors forcreating, editing, and viewing. In addition, each component may usededicated storage facilities. In the preferred embodiment, multimediadocuments are advantageously structured for authoring, storage, playbackand editing by storing some data under conventional file systems andsome data in special-purpose storage servers as will be discussed later.The Conventional File System 504 can be used to store allnon-time-sensitive portions of a multimedia document. In particular, thefollowing are examples of non-time-sensitive data that can be stored ina conventional type of computer file system:

-   -   1. structured and unstructured text    -   2. raster images    -   3. structured graphics and vector graphics (e.g., PostScript)    -   4. references to files in other file systems (video, hi-fidelity        audio, etc.) via pointers    -   5. restricted forms of executables    -   6. structure and timing information for all of the above        (spatial layout, order of presentation, hyperlinks, temporal        synchronization, etc.)

Of particular importance in multimedia documents is support fortime-sensitive media and media that have synchronization requirementswith other media components. Some of these time-sensitive media can bestored on conventional file systems while others may requirespecial-purpose storage facilities.

Examples of time-sensitive media that can be stored on conventional filesystems are small audio files and short or low-quality video clips(e.gas might be produced using QuickTime or Video for Windows). Otherexamples include window event lists as supported by the Window-EventRecord and Play system 512 shown in FIG. 30. This component allows forstoring and replaying a user's interactions with application programs bycapturing the requests and events exchanged between the client programand the window system in a time-stamped sequence. After this “record”phase, the resulting information is stored in a conventional file thatcan later be retrieved and “played” back. During playback the samesequence of window system requests and events reoccurs with the samerelative timing as when they were recorded. In prior-art systems, thiscapability has been used for creating automated demonstrations. In thepresent invention it can be used, for example, to reproduce annotatedsnapshots as they occurred at recording

As described above in connection with collaborative workstationsoftware, Snapshot Share 514 shown in FIG. 30 is a utility used inmultimedia calls and conferencing for capturing window or screensnapshots, sharing with one or more call or conference participants, andpermitting group annotation, telepoiriting, and re-grabs. Here, thisutility is adapted so that its captured images and window events can berecorded by the Window-Event Record and Play system 512 while being usedby only one person. By synchronizing events associated with a video oraudio stream to specific frame numbers or time codes, a multimedia callor conference can be recorded and reproduced in its entirety. Similarly,the same functionality is preferably used to create multimedia mailwhose authoring steps are virtually identical to participating in amultimedia call or conference (though other forms of MMM are notprecluded).

Some time-sensitive media require dedicated storage servers in order tosatisfy real-time requirements. High-quality audio/video segments, forexample, require dedicated real-time audio/video storage servers. Apreferred embodiment of such a server will be described later. Next tobe considered is how the current invention guarantees synchronizationbetween different media components.

Media Synchronization

A preferred manner for providing multimedia synchronization in thepreferred embodiment will next be considered. Only multimedia documentswith real-time material need include synchronization functions andinformation Synchronization for such situations may be provided asdescribed below.

Audio or video segments can exist without being accompanied by theother. If audio and video are recorded simultaneously (“co-recorded”),the preferred embodiment allows the case where their streams arerecorded and played back with automatic synchronization—as would resultfrom conventional VCRs, laserdisks, or time-division multiplexed(“interleaved”) audio/video streams. This excludes the need to tightlysynchronize (i.e., “lip-sync”) separate audio and video sequences.Rather, reliance is on the co-recording capability of the Real-TimeAudio/Video Storage Server 502 to deliver all closely synchronized audioand video directly at its signal outputs.

Each recorded video sequence is tagged with time codes (e.g. SMPTE at1/30 second intervals) or video frame numbers. Each recorded audiosequence is tagged with time codes (e.g., SMPTE or MIDI) or, ifco-recorded with video, video frame numbers. The preferred embodimentalso provides synchronization between window events and audio and/orvideo streams. The following functions are supported:

-   -   1. Media-time-driven Synchronization: synchronization of window        events to an audio, video, or audio/video stream, using the        real-time media as the timing source.    -   2. Machine-time-driven-Synchronization:        -   a. synchronization of window events to the system clock        -   b. synchronization of the start of an audio, video, or            audio/video segment to the system clock

If no audio or video is involved, machine-time-driven synchronization isused throughout the document Whenever audio and/or video is playing,media-time-synchronization is used. The system supports transitionbetween machine-time and media-time synchronization whenever anaudio/video segment is started or stopped.

As an example, viewing a multimedia document might proceed as follows:

-   -   Document starts with an annotated share (machine-time-driven        synchronization).    -   Next, start audio only (a “voice annotation”) as text and        graphical annotations on the share continue (audio is timing        source for window events).    -   Audio ends, but annotations continue (machine-time-driven        synchronization).    -   Next, start co-recorded audio/video continuing with further        annotations on same share (audio is timing source for window        events).    -   Next, start a new share during the continuing audio/video        recording; annotations happen on both shares (audio is timing        source for window events).    -   Audio/video stops, annotations on both shares continue        (machine-time-driven synchronization).    -   Document ends.

Audio/Video Storage

As described above, the present invention can include manyspecial-purpose servers that provide storage of time-sensitive media(e.g. audio/video streams) and support coordination with other media.This section describes the preferred embodiment for audio/video storageand recording services.

Although storage and recording services could be provided at each CMW,it is preferable to employ a centralized server 502 coupled to MLAN 10,as illustrated in FIG. 31. A centralized server 502, as shown in FIG.31, provides the following advantages:

-   -   1. The total amount of storage hardware required can be far less        (due to better utilization resulting from statistical        averaging).    -   2. Bulky and expensive compression/decompression hardware can be        pooled on the storage servers and shared by multiple clients. As        a result, fewer compression/decompression engines of higher        performance are required than if each workstation were equipped        with its own compression/decompression hardware.    -   3. Also, more costly centralized codecs can be used to transfer        mail wide area among campuses at far lower costs than attempting        to use data WAN technologies.    -   4. File system administration (e.g. backups and file system        replication, etc.) are far less costly and higher performance.

The Real-Time Audio/Video Storage Server 502 shown in FIG. 31Astructures and manages the audio/video files recorded and stored on itsstorage devices Storage devices may typically includecomputer-controlled VCRs, as well as rewritable magnetic or opticaldisks. For example, server 502 in FIG. 31A includes disks 60 e forrecording and playback. Analog information is transferred between disks60 e and the. A/V Switching Circuitry 30 via analog I/O 62 Control isprovided by control 64,coupled to Data LAN hub 25.

At a high level, the centralized audio/video storage and playback server502 in FIG. 31A performs the following functions:

-   -   1. File Management        -   It provides mechanisms for creating, naning, time-stamping,            storing, retrieving copying, deleting, and playing back some            or all portions of an audio/video file.    -   2. File Transfer and Replication        -   The audio/video file server supports replication of files on            different disks managed by the same file server to            facilitate simultaneous access to the same files. Moreover,            file transfer facilities are provided to support            transmission of audio/video files between itself and other            audio/video storage and playback engines. File transfer can            also be achieved by using the underlying audio/video network            facilities: servers establish a real-time audio/video            network connection between themselves so one server can            “play back” a file while the second server simultaneously            records it.    -   3. Disk Management        -   The storage facilities support specific disk allocation,            garbage collection and defragmentation facilities. They also            support mapping disks with other disks (for replication and            staging modes, as appropriate) and mapping disks, via I/O            equipment, with the appropriate Video/Audio network port.    -   4. Synchronization support        -   Synchronization between audio and video is ensured by the            multiplexing scheme used by the storage media, typically by            interleaving the audio and video streams in a            time-division-multiplexed fashion. Further, if            synchronization is required with other stored media (such as            window system graphics), then frame numbers, time codes, or            other timing events are generated by the storage server. An            advantageous way of providing this synchronization in the            preferred embodiment is to synchronize record and playback            to received frame number or time code events.    -   5. Searching        -   To support intra-file searching, at least start, stop,            pause, fast forward, reverse, and fast reverse operations            are provided. To support inter-file searching, audio/video            tagging, or more generalized “go-to” operations and            mechanisms, such as frame numbers or time code, are            supported at a search-function level.    -   6. Connection Management        -   The server handles requests for audio/video network            connections from client programs (such as video viewers and            editors running on client workstations) for real-time            recording and real-time playback of audio/video files.

Next to be considered is how centralized audio/video storage serversprovide for real-time recording and playback of video streams.

Real-Time Disk Delivery

To support real-time audio/video recording and playback, the storageserver needs to provide a real-time transmission path between thestorage medium and the appropriate audio/video network port for eachsimultaneous client accessing the server. For example, if one user isviewing a video file at the same time several other people are creatingand storing new video files on the same disk, multiple simultaneouspaths to the storage media are required. Similarly, video mail sent tolarge distribution groups, video databases, and similar functions mayalso require simultaneous access to the same video files, again imposingmultiple access requirements on the video storage capabilities.

For storage servers that are based on computer-controlled VCRs orrewritable laserdisks, a real-time transmission path is readilyavailable through the direct analog connection between the disk or tapeand the network port. However, because of this single direct connection,each VCR or laserdisk can only be accessed by one client program at thesame time (multi-head laserdisks are an exception). Therefore, storageservers based on VCRs and laserdisks are difficult to scale for multipleaccess usage. In the preferred embodiment, multiple access to the samematerial is provided by fie replication and staging, which greatlyincreases storage requirements and the need for moving informationquickly among storage media units serving different users.

Video systems based on magnetic disks are more readily scalable forsimultaneous use by multiple people. A generalized hardwareimplementation of such a scalable storage and playback system-502 isillustrated in FIG. 32. Individual I/O cards 530 supporting digital andanalog I/O are linked by intra-chassis digital networking (e.g. buses)for fie transfer within chassis 532 holding some number of these cardsMultiple chassis 532 are linked by inter-chassis networking. The DigitalVideo Storage System available from Parallax Graphics is an example ofsuch a system implementation.

The bandwidth available for the transfer of files among disks isultimately limited by the bandwidth of these intra-chassis andinter-chassis networking or systems that use sufficiently powerful videocompression schemes, real-time delivery requirements for a small numberof users can be met by existing file system software (such as the Unixfile system), provided that the block-size of the storage system isoptimized for video storage and that sufficient buffering is provided bythe operating system software to guarantee continuous flow of theaudio/video data.

Special-purpose software/hardware solutions can be provided to guaranteehigher performance under heavier usage or higher bandwidth conditions,for example, a higher throughput version of FIG. 32 is illustrated inFIG. 33, which uses crosspoint switching, such as provided by SCSICrossbar 540, which increases the total bandwidth of the inter-chassisand intra-chassis network, thereby increasing the number of possiblesimultaneous file transfers.

Real-Time Network Delivery

By using the same audio/video format as used for audio/videoteleconferencing, the audio/video storage system can leverage thepreviously described network facilities: the MLANs 10 can be used toestablish a multimedia network connection between client workstationsand the audio/video storage servers Audio/Video editors and viewersrunning on the client workstation use the same software interfaces asthe multimedia teleconferencing system to establish these networkconnections.

The resulting architecture is shown in FIG. 31B Client workstations usethe existing audio/video network to connect to the storage server'snetwork ports. These network ports are connected tocompression/decompression engines that plug into the server bus. Theseengines compress the audio/video streams that come in over the networkand store them on the local disk. Similarly, for playback, the serverreads stored video segments from its local disk and routes them throughthe decompression engines back to client workstations for local display.

The present invention allows for alternative delivery strategies orexample, some compression algorithms are asymmetric, meaning thatdecompression requires much less compute power than compression. In somecases, real-time decompression can even be done in software, withoutrequiring any special-purpose decompression hardware. As a result, thereis no need to decompress stored audio and video on the storage serverand play it back in realtime over the network. Instead, it can be moreefficient to transfer an entire audio/video file from the storage serverto the client workstation, cache it on the workstation's disk, and playit back locally. These observations lead to a modified architecture aspresented in FIG. 31C. In this architecture, clients interact with thestorage server as follows:

-   -   To record video, clients set up real-time audio/video network        connections to the storage server as before (this connection        could make use of an analog line).    -   In response to a connection request, the storage server        allocates a compression module to the new client.    -   As soon as the client starts recording, the storage server        routes the output from the compression hardware to an        audio/video file allocated on its local storage devices.    -   For playback, this audio/video file gets transferred over the        data network to the client workstation and pre-staged on the        workstation's local disk.    -   The client uses local decompression software and/or hardware to        play back the audio/video on its local audio and video hardware.

This approach frees up audio/video network ports andcompression/decompression engines on the server. As a result, the serveris scaled to support a higher number of simultaneous recording sessions,thereby further reducing the cost of the system. Note that such anarchitecture can be part of a preferred embodiment for reasons otherthan compression/decompression asymmetry (such as the economics of thetechnology of the day, existing embedded base in the enterprise, etc.).

Multimedia Conference Recording

Multimedia conference recording (MMCR) will next be considered. Forfull-feature multimedia desktop calls and conferencing (e.g. audio/videocalls or conferences with snapshot share), recording (storage)capabilities are preferably provided for audio and video of all parties,and also for all shared windows, including any telepointing andannotations provided during the teleconference. Using the multimediasynchronization facilities described above, these capabilities areprovided in a way such that they can be replayed with accuratecorrespondence in time to the recorded audio and video, such as bysynchronizing to frame numbers or time code events.

A preferred way of capturing audio and video from calls would be torecord all calls and conferences as if they were multi-party conferences(even for two-party calls), using video mosaicing, audio mixing andcut-and-pasting, as previously described in connection with FIGS. 7-11.It will be appreciated that MMCR as described will advantageously permitusers at their desktop to review real-time collaboration as itpreviously occurred, including during a later teleconference. The outputof a MMCR session is a multimedia document that can be stored, viewed,and edited using the multimedia document facilities described earlier.

FIG. 31D shows how conference recording relates to the various systemcomponents described earlier. The Multimedia Conference Record/Playsystem 522 provides the user with the additional GUIs (graphical userinterfaces) and other functions required to provide the previouslydescribed MMCR functionality.

The Conference Invoker 518 shown in FIG. 31D is a utility thatcoordinates the audio/video calls that must be made to connect theaudio/video storage server 502 with special recording outputs onconference bridge hardware (35 in FIG. 3). The resulting recording islinked to information identifying the conference, a function alsoperformed by this utility.

Multimedia Mail

Now considering multimedia mail (MMM), it will be understood that MMMadds to the above-described MMCR the capability of delivering delayedcollaboration, as well as the additional ability to review theinformation multiple times and, as described hereinafter, to edit,re-send, and archive it. The captured information is preferably asuperset of that captured during MMCR, except that no other user isinvolved and the user is given a chance to review and edit beforesending the message.

The Multimedia Mail system 524 in FIG. 31D provides the user with theadditional GUIs and other functions required to provide the previouslydescribed MMM functionality Multimedia Mail relies on a conventionalEmail system 506 shown in FIG. 31D for creating, transporting, andbrowsing messages. However, multimedia document editors and viewers areused for creating and viewing message bodies. Multimedia documents (asdescribed above) consist of time-insensitive components and timesensitive components. The Conventional Email system 506 relies on theConventional File system 504 and Real-Time Audio/Video Storage Server502 for storage support. The time insensitive components are transportedwithin the Conventional Email system 506, while the real-time componentsmay be separately transported through the audio/video network using filetransfer utilities associated with the Real-Time Audio/Video StorageServer 502.

Multimedia Document Management

Multimedia document management (MMDM) provides long-term, high-volumestorage for MMCR and MMM. The MMDM system assists in providing thefollowing capabilities to a CMW user:

-   -   1. Multimedia documents can be authored as mail in the MMM        system or as call/conference recordings in the MMCR system and        then passed on to the MMDM system.    -   2. To the degree supported by external compatible multimedia        editing and authoring systems, multimedia documents can also be        authored by means other than MMM and MMCR.    -   3. Multimedia documents stored within the MMDM system can be        reviewed and searched.    -   4. Multimedia documents stored within the MMDM system can be        used as material in the creation of subsequent MMM.    -   5. Multimedia documents stored within the MMDM system can be        edited to create other multimedia documents.

The Multimedia Document Management system 526 in FIG. 31D provides theuser with the additional GUIs and other functions required to providethe previously described MMDM functionality. The MMDM includessophisticated searching and editing capabilities in connection with theMMDM multimedia document such that a user can rapidly access desiredselected portions of a stored multimedia document. The SpecializedSearch system 520 in FIG. 31D comprises utilities that allow users to domore sophisticated searches across and within multimedia documents. Thisincludes context-based and content-based searches (employing operationssuch as speech and image recognition, information filters, etc.),time-based searches, and event-based searches (window events, callmanagement events, speech/audio events, etc.).

Classes of Collaboration

The resulting multimedia collaboration environment achieved by theabove-described integration of audio/video/data teleconferencing, MMCR,MMM and MMDM is illustrated in FIG. 34. It will be evident that eachuser can collaborate with other users in real-time despite separationsin space and time. In addition, collaborating users can accessinformation already available within their computing and informationsystems, including information captured from previous collaborations.Note in FIG. 34 that space and time separations are supported in thefollowing ways:

-   -   1. Same time, different place        -   Multimedia calls and conferences    -   2. Different time, same place        -   MMDM access to stored MMCR and MMM information, or use of            MMM directly (i.e., copying mail to oneself)    -   3. Different time, different place        -   MMM    -   4. Same time, same place        -   Collaborative, face-to-face, multimedia document creation

By use of the same user interfaces and network functions, the presentinvention smoothly spans these three venus.

Remote Access to Expertise

In order to illustrate how the present invention may be implemented andoperated, an exemplary preferred embodiment will be described havingfeatures applicable to the aforementioned scenario involving remoteaccess to expertise. It is to be understood that this exemplaryembodiment is merely illustrative, and is not to be considered aslimiting the scope of the invention, since the invention may be adaptedfor other applications (such as in engineering and manufacturing) oruses having more or less hardware, software and operating features andcombined in various ways.

Consider the following scenario involving access from remote sites to anin-house corporate “expert” in the trading of financial instruments suchas in the securities market:

The focus of the scenario revolves around the activities of a trader whois a specialist in securities. The setting is the start of his day athis desk in a major financial center (NYC) at a major U.S. investmentbank.

The Expert has been actively watching a particular security over thepast week and upon his arrival into the office, he notices it is on therise. Before going home last night, he previously set up his system tofilter overnight news on a particular family of securities and asecurity within that family. He scans the filtered news and sees a storythat may have a long-term impact on this security in question. Hebelieves he needs to act now in order to get a good price on thesecurity. Also, through filtered mail, he sees that his counterpart inLondon, who has also been watching this security, is interested ingetting our Expert's opinion once he arrives at work.

The Expert issues a multimedia mail message on the security to the headof sales worldwide for use in working with their client base. Also amongthe recipients is an analyst in the research department and hiscounterpart in London. The Expert, in preparation for his previouslyestablished “on-call” office hours, consults with others within thecorporation (using the videoconferencing and other collaborativetechniques described above), accesses company records from his CMW, andanalyzes such information, employing software-assisted analytictechniques. His office hours are now at hand, so he enters “intercom”mode, which enables incoming calls to appear automatically (withoutrequiring the Expert to “answer his phone” and elect to accept or rejectthe call).

The Expert's computer beeps, indicating an incoming call, and the imageof a field representative 201 and his client 202 who are located at abank branch somewhere in the U.S. appears in video window 203 of theExpert's screen (shown in FIG. 35). Note that, unless the call isconvened to a “conference” call (whether explicitly via a menu selectionor implicitly by calling two or more other participants or adding athird participant to a call), the callers will see only each other inthe video window and will not see themselves as part of a video mosaic.

Also illustrated on the Expert's screen in FIG. 35 is the CollaborationInitiator window 204 from which the Expert can (utilizing CollaborationInitiator software module 161 shown in FIG. 20) initiate and controlvarious collaborative sessions. For example, the user can initiate witha selected participant a video call (CALL button) or the addition ofthat selected participant to an existing video call (ADD button), aswell as a share session (SHARE button) using a selected window or regionon the screen (or a blank region via the WHITEBOARD button forsubsequent annotation). The user can also invoke his MAIL software (MAILbutton) and prepare outgoing or check incoming Email messages (thepresence of which is indicated by a picture of an envelope in the dog'smouth in In Box icon 205), as well as check for “I called” messages fromother callers (MESSAGES button) left via the LEAVE WORD button in videowindow 203 Video window 203 also contains buttons from which many ofthese and certain additional features can be invoked, such as hanging upa video call (HANGUP button), putting a call on hold (HOLD button),resuming a call previously put on hold (RESUME button) or muting theaudio portion of a call (MUTE button). In addition, the user can invokethe recording of a conference by the conference RECORD button. Alsopresent on the Expert's screen is a standard desktop window 206containing icons from which other programs (whether or not part of thisinvention) can be launched.

Returning to the example, the Expert is now engaged in a videoconferencewith field representative 201 and his client 202. In the course of thisvideoconference, as illustrated in FIG. 36, the field representativeshares with the Expert a graphical image 210 (pie chart of clientportfolio holdings) of his client's portfolio holdings (by clicking onhis SHARE button, corresponding to the SHARE button in video window 203of the Expert's screen, and selecting that image from his screen,resulting in the shared image appearing in the Share window 211 of thescreen of all participants to the share) and begins to discuss theclient's investment dilemma. The field representative also invokes acommand to secretly bring up the client profile on the Expert's screen.

After considering this information, reviewing the shared portfolio andasking clarifying questions, the Expert illustrates his advice bycreating (using his own modeling software) and sharing a new graphicalimage 220 (FIG. 37) with the field representative and his client. Eitherparty to the share can annotate that image using the drawing tools 221(and the TEXT button, which permits typed characters to be displayed)provided within Share window 211, or “regrab” a modified version of theoriginal image (by using the REGRAB button), or remove all suchannotations (by using the CLEAR button of Share window 211), or “grab” anew image to share (by clicking on the GRAB button of Share window 211and selecting that new image from the screen). In addition, anyparticipant to a shared session can add a new participant by selectingthat participant from the rolodex or quick-dial list (as described abovefor video calls and for data conferencing) and clicking the ADD buttonof Share window 211. One can also save the shared image (SAVE button),load a previously saved image to be shared (LOAD button), or print animage (PRINT button).

While discussing the Expert's advice, field representative 201 makesannotations 222 to image 220 in order to illustrate his concerns. Whileresponding to the concerns of field representative 201, the Expert hearsa beep and receives a visual notice (New Call window 223) on his screen(not visible to the field representative and his client), indicating theexistence of a new incoming call and identifying the caller. At thispoint, the Expert can accept the new call (ACCEPT button), refuse thenew call (REFUSE button, which will result in a message being displayedon the caller's screen indicating that the Expert is unavailable) or addthe new caller to the Expert's existing call (ADD button). In this case,the Expert elects yet another option (not shown)—to defer the call andleave the caller a standard message that the Expert will call back in Xminutes (in this case 1 minute). The Expert then elects also to deferhis existing call, telling the field representative and his client thathe will call them back in 5 minutes, and then elects to return theinitial deferred call.

It should be noted that the Expert's act of deferring a call results notonly in a message being sent to the caller, but also in the caller'sname (and perhaps other information associated with the call, such asthe time the call was deferred or is to be resumed) being displayed in alist 230 (see FIG. 38) on the Expert's screen from which the call can bereinitiated. Moreover, the “state” of the call (e.g., the informationbeing shared) is retained so that it can be recreated when the call isreinitiated Unlike a “hold” (described above), deferring a call actuallybreaks the logical and physical connections, requiring that the entirecall be reinitiated by the Collaboration initiator and the AVNM asdescribed above.

Upon returning to the initial deferred call, the Expert engages in avideoconference with caller 231, a research analyst who is located 10floors up from the Expert with a complex question regarding a particularsecurity Caller 231 decides to add London expert 232 to thevideoconference (via the ADD button in Collaboration Initiator window204) to provide additional information regarding the factual history ofthe security Upon selecting the ADD button, video window 203 nowdisplays, as illustrated in FIG. 38, a video mosaic consisting of threesmaller images (instead of a single large image displaying only caller231) of the Expert 233, caller 231 and London expert 232.

During this videoconference, an urgent PRIORITY request (New Call window234) is received from the Expert's boss (who is engaged in a three-partyvideoconference call with two members of the bank's operationsdepartment and is attempting to add the Expert to that call to answer aquick question). The Expert puts his three-party video-conference onhold (merely by clicking the HOLD button in video window 203) andaccepts (via the ACCEPT button of New Call window 234) the urgent callfrom his boss, which results in the Expert being added to the boss'three-party videoconference call.

As illustrated in FIG. 39, video window 203 is now replaced with afour-person video mosaic representing a four-party conference callconsisting of the Expert 233, his boss 241 and the two members 242 and243 of the bank's operations department. The Expert quickly answers theboss' question and, by clicking on the RESUME button (of video window203) adjacent to the names of the other participants to the call onhold, simultaneously hangs up on the conference call with his boss andresumes his three-party conference call involving the securities issue,as illustrated in video window 203 of FIG. 40.

While that call was on hold, however, analyst 231 and London expert 232were still engaged in a two-way videoconference (with a blackenedportion of the video mosaic on their screens indicating that the Expertwas on hold) and had shared and annotated a graphical image 250 (seeannotations 251 to image 250 of FIG. 40) illustrating certain financialconcerns. Once the Expert resumed the call, analyst 231 added the Expertto the share session, causing Share window 211 containing annotatedimage 250 to appear on the Expert's screen Optionally, snapshot sharingcould progress while the video was on hold.

Before concluding his conference regarding the securities, the Expertreceives notification of an incoming multimedia mail message—e.g., abeep accompanied by the appearance of an envelope 252 in the dog's mouthin In Box icon 205 shown in FIG. 40. Once he concludes his call, hequickly scans his incoming multimedia mail message by clicking on In Boxicon 205, which invokes his mail software, and then selecting theincoming message for a quick scan, as generally illustrated in the toptwo windows of FIG. 2B. He decides it can wait for further review as thesender is an analyst other than the one helping on his securityquestion.

He then reinitiates (by selecting deferred call indicator 230, shown inFIG. 40) his deferred call with field representative 201 and his client202, as shown in FIG. 41. Note that the full state of the call is alsorecreated, including restoration of previously shared image 220 withannotations 222 as they existed when the call was deferred (see FIG.37). Note also in FIG. 41 that, having reviewed his only unread incomingmultimedia mail message, In Box icon 205 no longer shows an envelope inthe dog's mouth, indicating that the Expert currently has no unreadincoming messages.

As the Expert continues to provide advice and pricing information tofield representative 201, he receives notification of three prioritycalls 261-263 in short succession Call 261 is the Head of Sales for theChicago office Working at home, she had instructed her CMW to alert herof all urgent news or messages, and was subsequently alerted to thearrival of the Expert's earlier multimedia mail message Call 262 is anurgent international call Call 263 is from the Head of Sales in LosAngeles. The Expert quickly winds down and then concludes his call withfield representative 201.

The Expert notes from call indicator 262 that this call is not only aninternational call (shown in the top portion of the New Call window),but he realizes it is from a laptop user in the field in Central Mexico.The Expert elects to prioritize his calls in the following manner: 262,261 and 263. He therefore quickly answers call 261 (by clicking on itsACCEPT button) and puts that call on hold while deferring call 263 inthe manner discussed above. He then proceeds to accept the callidentified by international call indicator 262.

Note in FIG. 42 deferred call indicator 271 and the indicator for thecall placed on hold (next to the highlighted RESUME button in videowindow 203), as well as the image of caller 272 from the laptop in thefield in Central Mexico. Although Mexican caller 272 is outdoors and hasno direct access to any wired telephone connection, his laptop has twowireless modems permitting dial-up access to two data connections in thenearest field office (through which his calls were routed). The systemautomatically (based upon the laptop's registered service capabilities)allocated one connection for an analog telephone voice call (using hislaptop's built-in microphone and speaker and the Expert'scomputer-integrated telephony capabilities) to provide audioteleconferencing. The other connection provides control, dataconferencing and one-way digital video (i.e., the laptop user cannot seethe image of the Expert) from the laptop's built-in camera, albeit at avery slow frame rate (e.g., 3-10 small frames per second) due to therelatively slow dial-up phone connection.

It is important to note that, despite the limited capabilities of thewireless laptop equipment, the present invention accommodates suchcapabilities, supplementing an audio telephone connection with limited(i.e., relatively slow) one-way video and data conferencingfunctionality. As telephony and video compression technologies improve,the present invention will accommodate such improvements automatically.Moreover, even with one participant to a teleconference having limitedcapabilities, other participants need not be reduced to this “lowestcommon denominator.” For example, additional participants could be addedto the call illustrated in FIG. 42 as described above, and suchparticipants could have full videoconferencing, data conferencing andother collaborative functionality vis-a-vis one another, while havinglimited functionality only with caller 272.

As his day evolved, the off-site salesperson 272 in Mexico was notifiedby his manager through the laptop about a new security and becameconvinced that his client would have particular interest in this issue.The salesperson therefore decided to contact the Expert as shown in FIG.42. While discussing the security issues, the Expert again shares allcaptured graphs, charts, etc.

The salesperson 272 also needs the Expert s help on another issue. Hehas hard copy only of a client's portfolio and needs some advice on itscomposition before he meets with the client tomorrow. He says he willfax it to the Expert for analysis Upon receiving the fax—on his CMW, viacomputer-integrated fax—the Expert asks if he should either send theMexican caller a “QuickTime” movie (a lower quality compressed videostandard from Apple Computer) on his laptop tonight or send ahigher-quality CD via FedX tomorrow—the notion being that the Expert canproduce an actual video presentation with models and annotations invideo form. The salesperson can then play it to his client tomorrowafternoon and it will be as if the Expert is in the room. The Mexicancaller decides he would prefer the CD.

Continuing with this scenario, the Expert learns, in the course of hiscall with remote laptop caller 272, that he missed an important issueduring his previous quick scan of his incoming multimedia mail message.The Expert is upset that the sender of the message did not utilize the“video highlight” feature to highlight this aspect of the message. Thisfeature permits the composer of the message to define “tags” (e.g., byclicking a TAG button, not shown) during record time which are storedwith the message along with a “time stamp,” and which cause a predefinedor selectable audio and/or visual indicator to be played/displayed atthat precise point in the message during playback.

Because this issue relates to the caller that the Expert has on hold,the Expert decides to merge the two calls together by adding the call onhold to his existing call. As noted above, both the Expert and thepreviously held caller will have full video capabilities vis-a-vis oneanother and will see a three-way mosaic image (with the image of caller272 at a slower frame rate), whereas caller 272 will have access only tothe audio portion of this three-way conference call, though he will havedata conferencing functionality with both of the other participants.

The Expert forwards the multimedia mail message to both caller 272 andthe other participant, and all three of them review the video enclosurein greater detail and discuss the concern raised by caller 272. Theyshare certain relevant data as described above and realize that theyneed to ask a quick question of another remote expert. They add thatexpert to the call (resulting in the addition of a fourth image to thevideo mosaic, also not shown) for less than a minute while they obtain aquick answer to their question. They then continue their three-way calluntil the Expert provides his advice and then adjourns the call.

The Expert composes a new multimedia mail message, recording his imageand audio synchronized (as described above) to the screen displaysresulting from his simultaneous interaction with his CMW (e.g., runninga program that performs certain calculations and displays a graph whilethe Expert illustrates certain points by telepointing on the screen,during which time his image and spoken words are also captured). Hesends this message to a number of salesforce recipients whose identitiesare determined automatically by an outgoing mail filter that utilizes adatabase of information on each potential recipient (e.g., selectingonly those whose clients have investment policies which allow this typeof investment).

The Expert then receives an audio and visual reminder (not shown) that aparticular video feed (e.g., a short segment of a financial cabletelevision show featuring new financial instruments) will be triggeredautomatically in a few minutes. He uses this time to search his localsecurities database, which is dynamically updated from financialinformation feeds (e.g., prepared from a broadcast textual stream ofcurrent financial events with indexed headers that automatically appliesdata filters to select incoming events relating to certain securities).The video feed is then displayed on the Expert's screen and he watchesthis short video segment.

After analyzing this extremely up-to-date information, the Expert thenreinitiates his previously deferred call, from indicator 271 shown inFIG. 42, which he knows is from the Head of Sales in Los Angeles, who isseeking to provide his prime clients with securities advice on anothersecurities transaction based upon the most recent available information.The Expert's call is not answered directly, though he receives a shortprerecorded video message (left by the caller who had to leave his homefor a meeting across town soon after his priority message was deferred)asking that the Expert leave him a multimedia mail reply message withadvice for a particular client, and explaining that he will access thismessage remotely from his laptop as soon as his meeting is concluded.The Expert complies with this request and composes and sends this mailmessage.

The Expert then receives an audio and visual reminder on his screenindicating that his office hours will end in two minutes. He switchesfrom “intercom” mode to “telephone” mode so that he will no longer bedisturbed without an opportunity to reject incoming calls via the NewCall window described above. He then receives and accepts a final callconcerning an issue from an electronic meeting several months ago, whichwas recorded in its entirety.

The Expert accesses this recorded meeting from his “corporate memory.”He searches the recorded meeting (which appears in a second video windowon his screen as would a live meeting, along with standard controls forstop/play/rewind/fast forward/etc.) for an event that will trigger hismemory using his fast forward controls, but cannot locate the desiredportion of the meeting. He then elects to search the ASCII text log(which was automatically extracted in the background after the meetinghad been recorded, using the latest voice recognition techniques), butstill cannot locate the desired portion of the meeting. Finally, heapplies an information filter to perform a content-oriented (rather thanliteral) search and finds the portion of the meeting he was seeking.After quickly reviewing this short portion of the previously recordedmeeting, the Expert responds to the caller's question, adjourns the calland concludes his office hours.

It should be noted that the above scenario involves manystate-of-the-art desktop tools (e.g., video and information feeds,information filtering and voice recognition) that can be leveraged byour Expert during videoconferencing, data conferencing and othercollaborative activities provided by the present invention—because thisinvention, instead of providing a dedicated videoconferencing system,provides a desktop multimedia collaboration system that integrates intothe Expert's existing workstation/LAN/WAN environment.

It should also be noted that all of the preceding collaborativeactivities in this scenario took place during a relatively short portionof the expert's day (e.g., less than an hour of cumulative time) whilethe Expert remained in his office and continued to utilize the tools andinformation available from his desktop. Prior to this invention, such ascenario would not have been possible because many of these activitiescould have taken place only with face-to-face collaboration, which inmany circumstances is not feasible or economical and which thus may wellhave resulted in a loss of the associated business opportunities.

Although the present invention has been described in connection withparticular preferred embodiments and examples, it is to be understoodthat many modifications and variations can be made in hardware,software, operation, uses, protocols and data formats without departingfrom the scope to which the inventions disclosed herein are entitled.For example, for certain applications, it will be useful to provide someor all of the audio/video signals in digital form. Accordingly, thepresent invention is to be considered as including all apparatus andmethods encompassed by the appended claims.

1. A method of communication between a plurality of users, comprising:allowing at least first and second users to establish communication overat least two separate communications networks by first and secondcommunication devices, associated respectively with the first and secondusers, registering with at least one communications network; maintainingat least one service record for at least the first and second users,each service record including at least the second user's locationinformation; allowing the first user to request communication with thesecond user; causing retrieval of the location information of the seconduser in response to the request by the first user; initiating acommunication, including at least audio, between the first and secondusers by using at least one of the communications networks and theretrieved location information; and wherein the audio communication istransmitted digitally over at least one network.
 2. The method of claim1, wherein registration with at least one communication network iscaused by the user logging in at the respective communication device. 3.The method of claim 2, wherein at least the second users' locationinformation includes information identifying the user's location nomatter where that user is located.
 4. The method of claim 2, wherein theservice record further includes information usable to associate a userwith at least one port associated with that user's communication device.5. The method of claim 3, wherein the location information includesaddress information.
 6. The method of claim 3, wherein the communicationfurther includes video.
 7. The method of claim 3,.wherein the first userrequests communication with the second user by selecting an identifierfor the second user from a list of displayed user identifiers displayedto the first user.
 8. The method of claim 3, further comprisingindicating to the first user if another user has logged in.
 9. Themethod of claim 8, wherein indicating to the first user whether anotheruser is not logged in occurs if no service record is found for the otheruser.
 10. The method of claim 2, wherein the service record furtherincludes information as to the capabilities of at least one of thecommunication devices.
 11. The method of claim 10, wherein theinformation about the capabilities includes information about the typesof communications in which a device can participate.
 12. The method ofclaim 1, wherein at least one of the networks is a wireless network. 13.The method of claim 2, wherein at least one of the networks is a widearea network.
 14. The method of claim 11, wherein the service record forthe second user is maintained by an entity other than the network thesecond user uses to establish the communications.
 15. The method ofclaim 3, wherein the retrieved location information is provided to thecommunications device of the first user.
 16. The method of claim 3,wherein the service record is maintained by at least one communicationsnetwork.
 17. The method of claim 16, further comprising: detecting anattempt by the first user to initiate a communication with a third user;causing an indication of the attempt to initiate communications toappear automatically on third user's display; and allowing the thirduser to accept or reject the communication.
 18. The method of claim 1,wherein at least one of the service records is entered when theassociated user logs in to the network.
 19. The method of claim 5,further comprising creating at least first and second call handles,respectively associated with the first and second users' communicationdevices, each call handle defining at least a call state for itsrespective device.
 20. The method of claim 3, wherein the first user isassociated with a first communication network and the second user isassociated with a second communication network.
 21. A system forcommunicating between a plurality of users, comprising: (a) at least onecommunication device associated with each user; (b) at least onecommunications network over which at least first and second users canestablish communication by at least one communication device registeringwith at least one communication network; and (c) at least one servicerecord, for at least the second user, including at least the seconduser's location information; wherein the system is configured to: allowthe first user to request communication with the second user; retrievethe location information of the second user in response to the requestby the first user; and initiate a communication, including at leastaudio communication, between the first and second users by using atleast one of the communications networks and the retrieved locationinformation; and creating at least a first and second call handle,respectively associated with the first and second users' communicationdevices, each call handle defining at least a call state for itsassociated device.
 22. The system of claim 21, wherein registration withat least one communication network is caused by the user logging in atthe respective communication device.
 23. The system of claim 22, whereinat least the second users' location information includes informationidentifying the-user's location no matter where that user is located.24. The system of claim 23, wherein the communication further includesvideo.
 25. The system of claim 23, wherein the system is furtherconfigured to: cause a list of user identifiers to be displayed on thedisplay, the list including an identifier for at least the second user;and allow the first user to select the identifier for the second user tofacilitate retrieving the location information.
 26. The system of claim23, wherein the system is configured to indicate to the first user ifanother user has not logged in.
 27. The method of claim 4, whereinindicating to the first user whether another user is not logged inoccurs if no service record is found for the other user.
 28. The systemof claim 23, wherein the service record further includes information asto the capabilities of at least one of the communication devices. 29.The system of claim 27, wherein the information about the capabilitiesincludes information about the types of communications in which a devicecan participate.
 30. The system of claim 22, wherein the service recordfurther includes information usable to associate a user with at leastone port associated with that user's communication device.
 31. Thesystem of claim 21, wherein at least one of the networks is a wirelessnetwork.
 32. The system of claim 22, wherein at least one of thenetworks is a wide area network.
 33. The system of claim 23, wherein theservice record for the second user is maintained by an entity other thanthe network the second user uses to establish the communications. 34.The system of claim 23, wherein the retrieved location information isprovided to the communication device of the first user.
 35. The systemof claim 23, wherein the service record is maintained by at least onecommunications network.
 36. The system of claim 35, wherein the systemis configured to: (a) detect an attempt by the first user to initiate acommunication with a third user; (b) cause an indication of the attemptto initiate communications to appear automatically on the third user'sdisplay; and (c) allow the third user to accept or reject thecommunication.
 37. The system of claim 21, wherein at least one of theservice records is entered when the associated user logs in to thenetwork.
 38. The system of claim 25, wherein the system is furtherconfigured to initiate the communication by creating at least a firstand second call handle, respectively associated with the first andsecond users' communication devices, each call handle defining at leasta call state for its associated device.
 39. The method of claim 23,wherein the first user is associated with a first communication networkand the second user is associated with a second communication network.40. The method of claim 23, wherein the location information includesaddressing information.